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need  for  experimentation.  An  analysis  of  how  the  current  two-tiered  process, 
which  separates  technology  development  from  project-based  acquisition, 
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creates  a  disaggregated  knowledge  base,  which  exacerbates  the  difficulty  of 
top-down  specification  and  bottom-up  integration;  and  (c)  fails  to  generate 
an  entrepreneurial  supply-side  spirit.  Recommendations  for  improvement  are 
provided. 
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Cost  growth  in  defense  acquisition  is  both  a  problem  in  its  own  right  and  part 
of  the  larger  phenomenon  of  programs  that  fail  to  perform  as  intended  or 
desired.  It  is  a  limited  but  persistent  phenomenon,  which  has  not  improved 
in  any  material  respect  over  at  least  the  past  four  decades;  nor  is  it  unique  to 
defense,  and  it  can  flow  from  a  variety  of  causes.  A  limited  group  of  similar 
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lack  of  clear  analysis  of  underlying  causes.  Research  points  to  a  corrective 
technique,  “taking  the  outside  view,”  or  “reference  class  forecasting,”  with 
clear  promise  for  attacking  the  root  problems. 
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world’s  best  combat  capabilities  by  the  U.S.  Department  of  Defense  and  the 
defense  industry.  These  two  complex  enterprises,  on  behalf  of  the  nation  and 
its  allies,  are  actively  developing,  producing,  fielding,  and  sustaining  combat 
systems  for  joint  warfighters  that  are  second  to  none.  Does  this  shared  interac¬ 
tion  form  an  effective  partnership?  In  this  article,  the  authors  analyze  private 
industry’s  perception  of  the  challenges/opportunities  that  exist  in  the  shared 
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The  Defense  Acquisition  Workforce  is  getting  younger,  and  its  educational 
expectations  include  using  advanced  and  innovative  learning  technologies. 
The  Defense  Acquisition  University  (DAU)  has  fully  embraced  this  genera¬ 
tional  trend  and  has  partnered  with  several  institutions  to  conduct  research 
on  Advanced  Learning  Technologies,  or  ALT.  One  such  partnership  is  with 
George  Mason  University’s  Instructional  Technology  Immersion  Program.  The 
partnership's  goal  was  to  examine  DAU’s  current  learning  assets  and  identify 
processes  and  methods  for  utilizing  innovative  learning  technology  designs. 
This  article  summarizes  this  effort  and  describes  the  resulting  online  perfor¬ 
mance  support  tool  called  LATIST  (Learning  Asset  Technology  Integration 
Support  Tool)  developed  to  facilitate  the  understanding,  selection,  and  inte¬ 
gration  of  ALT  by  DAU  faculty  and  staff. 
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From  the  Chairman  and  Executive  Editor 


FROM  THE  CHAIRMAN  AND 
EXECUTIVE  EDITOR 


The  theme  of  this  issue  of  the  Defense 
Acquisition  Research  Journal  is  derived 
from  the  excellent  review  by  Eunice 
Maytorena  of  Megaprojects  and  Risk:  An 
Anatomy  of  Ambition  (Flyvbjerg,  Bru- 
zelius,  &  Rothengatter)  for  the  Defense 
Acquisition  Professional  Reading  List. 

The  book  examines  very  large  projects 
(characterized  by  major  expense,  complexity,  innovation,  and 
impact)  and  why  they  so  often  run  over  budget  and  schedule. 
Although  the  authors  look  at  civilian  projects,  their  findings 
on  the  role  of  risk  and  stakeholder  accountability  in  decision 
making  are  directly  applicable  to  major  defense  acquisition 
programs. 

Leading  this  issue,  Donald  Birchler  and  his  coauthors  look 
at  one  of  the  elements  of  cost  risk  in  defense  megaprojects— 
the  role  of  concurrency  in  the  processes  of  development  and 
procurement.  Zoe  Szajnfarber  and  her  coauthors  examine 
these  two  processes,  with  specific  reference  to  the  space  sec¬ 
tor,  through  the  lens  of  innovation,  identifying  shortfalls  in  the 
current  approaches,  and  suggesting  ways  to  improve  them. 
William  O’Neil  explores  the  use  of  “reference  class  forecast¬ 
ing,”  a  statistical  methodology  pioneered  by  Bent  Flyvbjerg 
(one  of  the  Megaprojects  authors)  to  attack  the  root  cause 
of  cost  growth. 

Steve  Mills  and  his  coauthors  analyze  the  partnerships 
between  two  mega-organizations:  the  U.S.  Department  of 
Defense  and  the  defense  industry.  Finally,  Nada  Dabbagh 
and  her  team  show  how  innovative  learning  technologies  can 
improve  the  training  of  the  Defense  Acquisition  Workforce, 
which  is  the  most  critical  element  of  any  megaproject  or 
mega-organization. 

Dr.  Larrie  D.  Ferreiro 
Executive  Editor 
Defense  ARJ 
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COST  IMPLICATIONS 
OF  DESIGN/BUILD 
CONCURRENCY 

f  Donald  Birchler,  Gary  Christie,  and  Eric  Groo 

Developing  a  weapon  while  in  production  does  increase 
program  risk  and  is  sometimes  cited  as  a  reason  for  cost  growth. 

This  article  explores  the  relationship  between  concurrency  and 
cost  growth  in  large  weapon  programs.  The  authors  defined 
concurrency  as  the  proportion  of  research,  development,  and 
test  and  evaluation  appropriations  authorized  during  the  same 
years  in  which  procurement  appropriations  are  authorized.  Their 
results  strongly  indicate  that  concurrency  does  not  necessarily 
predict  cost  growth.  Using  classical  regression  techniques, 
the  authors  found  no  evidence  supporting  this  relationship. 

To  investigate  other  relationships  between  cost  growth  and 
concurrency,  they  also  used  a  smooth  curving  technique. 

These  experiments  showed  that,  although  the  relationship  is 
not  strong,  low  levels  of  concurrency  are  more  problematic 
than  higher  levels. 


Keywords:  Weapon  System;  Cost  Growth; 
Concurrency;  Ordinary  Least  Squares  (OLS) 
Regression;  Research,  Development,  Test  and 
Evaluation  (RDT&E) 
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Typically,  defense  programs  experience  some  level  of  concur¬ 
rency;  that  is,  production  of  the  weapon  system  happens  while  some 
portions  of  the  design  are  still  being  completed.  Many  people  within 
the  defense  acquisition  community  argue  that  high  levels  of  design/ 
build  concurrency  ultimately  lead  to  cost  growth,  as  it  implicitly 
creates  a  greater  level  of  risk.  For  example,  a  memorandum  from 
the  Assistant  Secretary  of  the  Navy  for  Research,  Development  and 
Acquisition  (ASN-RDA)  identified  the  high  degree  of  concurrency 
in  the  Littoral  Combat  Ship  as  being  a  large  contributor  to  the  pro¬ 
gram’s  overall  cost  growth  (DoD,  2006). 

In  a  zero-risk  world,  the  requirements,  concept  of  operations, 
and  substantial  prior  development  would  be  completed  before  the 
release  of  the  Request  for  Proposal  (RFP)  for  the  design  phase.  In 
addition,  100  percent  of  the  design  would  be  complete  before  the 
release  of  the  production  RFP;  and  all  the  initial  material/compo¬ 
nents  would  always  be  procured  and  available  before  production 
started.  Moreover,  requirements  would  not  change  once  design 
started,  design  would  not  change  once  production  started,  and 
production  would  flow  smoothly  without  delays  caused  by  late  soft¬ 
ware  or  hardware.  Thus,  in  a  zero-risk  world  we  would  say  programs 
have  zero  overlap,  or  concurrency,  and  virtually  no  production  risk. 

Unfortunately,  this  zero-risk  approach  to  production  planning  is 
impossible  to  achieve,  and  even  if  it  were,  many  would  argue  that  it 
is  not  desirable.  The  Japanese,  for  example,  pioneered  the  “just-in- 
time”  inventory  strategy,  where  materials  essential  for  production 
are  not  only  unavailable  before  production  start,  they  are  deliber¬ 
ately  fabricated  and  delivered  at  the  last  possible  moment  to  reduce 
in-process  inventory,  thus  reducing  storage  and  finance  costs  asso¬ 
ciated  with  inventory  beyond  what  is  immediately  needed.  No 
financial  or  technical  reasons  preclude  production  in  one  portion  of 
the  program  while  a  design  is  completed  on  an  unrelated  portion. 

Other  reasons  to  inject  plans  with  some  design/build  concur¬ 
rency,  despite  potential  increases  in  risk  of  cost  growth,  include  (a) 
urgent  need  for  the  product,  (b)  maintaining  the  industrial  base,  (c) 
avoiding  obsolescence,  and  (d)  reducing  exposure  to  requirements 
changes. 

Consequently,  major  programs  always  retain  some  level  of  con¬ 
currency,  much  of  which  is  actually  an  integral  part  of  the  plan  (see 
[a]  through  [c],  previous  paragraph). 

In  sum,  there  seems  to  be  a  good  case  to  be  made  that  concur¬ 
rency  is  actually  desirable  and  possibly  reduces  cost,  and  another 
equally  good  case  that  it  adds  risk,  which  ultimately  leads  to  cost 
growth.  Unfortunately,  despite  decades  of  interest  in  concurrency 
within  the  acquisition  community,  the  literature  on  concurrency 
is  surprisingly  thin.  We  found  only  one  study  conducted  by  the 


Cost  Implications  of  Design/Build  Concurrency 


July  2011 


Congressional  Budget  Office  (1988)  that  specifically  looked  at  the 
correlation  between  concurrency  and  cost  growth.  Another  more 
recent  study  by  RAND  touched  on  concurrency,  but  was  primarily 
about  other  factors  that  lead  to  cost  growth  (Arena,  Leonard,  Mur¬ 
ray,  &  Younossi,  2006).  However,  in  both  cases,  the  studies  found 
only  a  very  weak  correlation  between  concurrency  (defined  as 
the  overlap  between  operational  test  and  evaluation  and  produc¬ 
tion)  and  cost  growth.  Our  study,  done  on  behalf  of  the  ASN-RDA, 
examines  this  relationship  in  more  detail  using  a  slightly  different 
definition  of  concurrency  and  a  larger  data  set. 


Definitions 

In  general,  a  lack  of  consensus  prevails  regarding  the  meaning 
of  concurrency  in  acquisition  programs.  We  chose  a  definition  that 
reflects  the  most  general  use  of  concurrency  and  was  tractable  for 
analysis  given  the  data  available  for  large  acquisition  programs. 
Other  definitions  for  concurrency  exist  and  likely  have  different 
implications  in  those  contexts.  Our  definitions  for  concurrency  and 
cost  growth  follow: 

•  Concurrency  is  the  proportion  of  research,  develop¬ 
ment,  test  and  evaluation  (RDT&E)  appropriations  that 
are  authorized  during  the  same  years  that  procure¬ 
ment  appropriations  are  authorized.  This  proportion  is 
further  restricted  to  the  first  95  percent  of  total  RDT&E 
spending. 

•  Cost  growth  is,  after  adjusting  for  quantity  changes  and 
inflation,  the  proportional  increase  of  the  final  cost  to 
the  initial  cost  estimate. 

We  chose  95  percent  of  the  total  RDT&E  appropriation  because 
RDT&E  monies  continue  throughout  the  life  of  the  program,  albeit  at 
a  much  reduced  rate  toward  the  design-complete/testing-complete 
phase  of  the  program.  This  is  usually  due  to  the  ongoing  need  for 
updates  and  modifications,  but  has  little  bearing  on  concurrency 
issues.  We  were  satisfied,  after  a  little  experimentation,  that  the  95 
percent  cutoff  addressed  this  for  most  programs. 

This  measure  is  not  a  perfect  proxy  for  concurrency.  If  anything, 
the  95  percent  cutoff  likely  overstates  concurrency.  Moreover,  it  also 
misses  concurrency  in  related  programs  that  can  have  a  significant 
effect  on  cost  and  schedule  of  an  item  such  as  concurrency  of 
weapon  production  with  development  of  items  designated  for  the 
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weapon,  but  being  developed  under  other  programs  (such  as  radar, 
sonar,  etc.,  which  are  being  developed  for  more  than  one  platform). 

The  definition  for  cost  growth  may  initially  appear  to  be  overly 
broad,  allowing  for  the  inclusion  of  costs  that  are  completely  unre¬ 
lated  to  concurrency.  However,  adjustments  for  these  costs  would 
have  been  much  more  complex,  requiring  systemic  changes  in  both 
the  initial  estimates  and  final  profiles  tailored  to  each  program. 
Out  of  concern  that  this  process  would  become  ad  hoc,  we  left  the 
definition  broadly  defined  with  adjustments  made  for  quantity  and 
inflation  only. 


Data 

For  measures  of  cost  growth  and  concurrency,  we  gathered  data 
from  Selected  Acquisition  Reports  (SARs)  on  the  procurement  and 
RDT&E  profiles.  The  SARs  are  available  in  the  Defense  Acquisition 
Management  Information  Retrieval  System  (DAMIR).  We  reviewed 
this  list  and  selected  programs  based  on  their  maturity  and  avail¬ 
ability  of  data. 

On  some  occasions,  we  needed  to  drop  new  lines  of  produc¬ 
tion  from  final  SARs  that  were  absent  from  the  first  SAR.  This  was 
necessary  to  make  apples-to-apples  comparisons,  particularly 
when  controlling  for  quantity  changes,  due  to  the  tendency  of 
some  programs  to  add  on  additional  lines  of  production  to  existing 
procurement  programs. 

An  illustrative  case  is  the  V- 22.  The  initial  estimate  for  the  pro¬ 
gram  was  essentially  for  a  single  airframe  for  use  by  the  Marines. 
During  the  course  of  the  program,  the  Air  Force  Special  Forces 
ordered  a  modified  version  of  the  airframe.  This  new  line  of  pro¬ 
duction,  however,  cost  dramatically  more  than  the  Marine  version, 
presumably  because  of  modifications  and  enhancements  neces¬ 
sitated  by  the  requirements  for  Special  Forces  operations.  This 
growth  in  unit  costs  was  obviously  due  to  scope  changes  and  not 
incidental  to  changes  in  program  quantity  or  concurrency— our  pri¬ 
mary  controls  in  this  study.  Fortunately,  the  additional  RDT&E  and 
procurement  costs  associated  with  these  units  were  entirely  funded 
out  of  Air  Force  appropriations,  making  it  relatively  easy  to  exclude 
these  costs  from  the  cost  growth  and  concurrency  calculations.  For 
other  programs  with  similar  issues,  where  the  distinction  was  less 
apparent,  we  reviewed  budget  exhibits  and  other  publicly  available 
budget  justification  materials  for  information  to  tease  new  subpro¬ 
grams  away  from  historical  program  plans.  This  was  not  always 
possible,  leading  us  to  drop  several  programs  from  the  analysis. 
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Method 

Our  approach  was  driven  by  two  primary  questions: 

•  Relative  to  cost  growth,  is  there  an  ideal  amount  of 
concurrency  that  should  be  programmed  for  large 
acquisitions? 

•  If  there  is  no  “ideal,”  what  is  the  relationship,  if  any, 
between  cost  growth  and  concurrency? 

These  questions  suggested  a  hybrid  approach,  employing  tra¬ 
ditional  statistics  and  hypothesis  testing  methods  as  well  as  more 
modern  methods  of  data  exploration.  First,  using  Ordinary  Least 
Squares  (OLS)  regression,  we  fit  a  global  quadratic  function  to  the 
data.  The  quadratic  model  did  not  fit  the  data  well,  which  led  us  to 
consider  a  second  approach— locally  weighted  scatterplot  smooth¬ 
ing  (LOESS).  LOESS  is  a  nonparametric  regression  method,  which 
allows  the  data  to  express  and  inform  without  restricting  the  data  to 
fit  some  function.  We  assessed  the  results  of  this  second  approach 
with  a  bootstrapping  technique  (Efron  &  Tibshirani,  1998). 

To  ensure  that  we  were  using  completed  cost  growth  profiles, 
we  sampled  from  mature  programs,  defined  as  programs  that  had 
begun  Initial  Operating  Capability,  contained  in  DAMIR.  Of  these, 
after  discarding  programs  for  which  we  were  unable  to  locate  ini¬ 
tial  baseline  cost  estimates,  we  were  left  with  an  initial  set  of  43 
programs.  For  these  complete  programs,  we  used  the  procure¬ 
ment  and  RDT&E  acquisition  profiles  to  calculate  cost  growth  and 
concurrency. 

To  facilitate  making  statistical  inferences  about  concurrency,  we 
first  needed  to  directly  control  a  few  known,  significant  influences. 
First,  to  control  changes  in  base  years  dollars  between  SARs,  we 
rebaselined  all  the  reported  costs  in  constant  2009  dollars  using 
the  appropriate  inflation  indices  in  the  National  Defense  Budget 
Estimates,  commonly  referred  to  as  the  “Green  Book.”1  The  Green 
Book  published  indices  only  out  to  2014,  so  to  adjust  programs  that 
were  funded  past  this  year,  we  extended  the  indices  at  a  fixed  rate 
of  2  percent  per  year. 

Second,  we  needed  to  adjust  procurement  cost  growth  to 
reflect  changes  in  quantity.  When  the  first  SAR  is  published,  a  pro¬ 
curement  profile  shows  how  many  units  will  be  purchased  and  in 
what  years  they  will  be  purchased.  The  amount  of  units  purchased 
affects  the  procurement  cost  estimates  via  the  learning  curve  effect. 
That  is,  the  marginal  costs  of  production  drop  with  quantity  as,  with 
each  additional  unit,  workers  become  more  efficient,  manufacturing 
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processes  are  refined,  and  quality  control  improves.  This  process  is 
incorporated  into  every  baseline  cost  estimate. 

Thus,  it  is  important  to  adjust  the  original  cost  estimates 
reflected  in  the  first  SAR  to  account  for  the  changes  in  the  quan¬ 
tity  procured.  For  example,  a  program  that  was  originally  going 
to  purchase  100  units  at  a  total  procurement  cost  of  $1,000  faces 
a  budget  cut  by  Congress  leading  to  only  50  units  being  bought. 
A  new  baseline  estimate  could  be  calculated  by  simply  taking  the 
average  cost  per  unit  (i.e.,  $10)  and  subtracting  these  50  units  out 
of  the  procurement  funds.  Using  that  method,  we  would  simply 
multiply  50  units  by  $10  to  get  an  adjusted  original  cost  baseline 
of  just  $500. 

But  that  is  not  satisfactory.  In  fact,  by  not  buying  the  other  50 
units,  the  program  does  not  experience  the  same  level  of  learn¬ 
ing,  and  the  average  cost  per  unit  actually  rises  as  a  result.  In  our 
example  cited  previously,  the  average  cost  per  unit  would  rise  to 
something  over  $10,  and  the  procurement  savings  would  be  less 
than  $500.  Using  a  technique  pioneered  by  Goldberg  and  Touw 
(2003),  we  were  able  to  estimate  the  learning  curve  effects  for  the 
programs  in  our  data  set  and  adjust  the  original  cost  baseline  up  or 
down,  depending  on  whether  fewer  or  more  units  were  procured. 

Figure  1  illustrates  the  learning  curve  adjustment  for  the  F-22 
program.  The  gray  squares  correspond  to  the  quantities  and  costs 
reported  in  the  first  SAR.  Notice  that  the  gold  squares  curve  sharply 
downward,  but  then  flatten  out  as  the  total  quantity  increases.  This 
pattern  corresponds  to  an  anticipated  initial  period  of  intensive 
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learning,  which  progressively  tapers  as  the  gains  from  learning  dis¬ 
appear.  The  gold  line  is  the  estimated  learning  curve.  What  is  most 
striking  about  the  line  is  how  closely  it  appears  to  fit  the  data,  with¬ 
out  any  additional  modification. 

Reducing  the  procurement  quantities  increases  the  average 
costs  of  the  units  purchased,  as  the  lower  cost  units  at  the  end  of 
the  production  run  are  not  added  into  the  total  production  run. 
Thus,  if  the  program  had  followed  the  initial  learning  curve,  then 
the  lot  average  cost  would  have  fallen  along  the  upper  portion  of 
the  initial  estimate.  These  adjusted  lot  average  costs,  indicated  by 
the  gray  line,  form  the  new  baseline  for  measuring  cost  growth.  The 
gray  squares  correspond  to  the  actual  quantities  and  costs  reported 
in  the  final  SAR  profile  for  the  F-22  program.  Despite  higher  than 
expected  total  costs,  the  average  unit  costs  decline  at  a  rate  reflec¬ 
tive  of  the  original  estimate.  Comparing  the  gray  squares  to  the 
gray  line,  we  can  measure  cost  growth  as  the  difference  between 
the  adjusted  initial  estimate  and  the  final  reported  cost  profile  for 
a  program.  This  is  literally  the  area  demarcated  by  the  horizontal 
dotted  lines. 

Most  of  the  programs  that  we  examined  experienced  some 
change  to  the  procurement  quantities.  This  adjustment  required 
stable  associations  between  procurement  costs  and  units  for  pro¬ 
grams  between  the  first  and  final  cost  profiles.  Unfortunately,  this 
requirement  reduced  the  data  set  to  only  28  programs  suitable  for 
analysis  (Table  1). 


Results 

For  procurement  cost  growth,  we  wanted  to  see  if  there  was  any 
correlation  to  concurrency,  as  measured  by  the  percent  of  RDT&E 
spending  that  occurs  when  procurement  spending  is  happening  at 
the  same  time.  As  mentioned  in  the  method  section  of  this  article, 
we  calculated  concurrency  in  two  ways.  First,  we  used  the  first  pub¬ 
lished  SAR  to  determine  planned  concurrency.  We  then  used  the 
last  SAR  to  calculate  actual  concurrency.  Thus,  for  each  element  of 
cost  growth,  we  looked  for  correlations  with  two  different  measures 
of  concurrency. 

Based  upon  the  feedback  that  we  received  from  various  Navy 
and  DoD  acquisition  officials,  we  decided  that  a  good  starting 
hypothesis  was  that  concurrency  follows  a  Goldilocks  rule  (not  too 
much,  not  too  little,  but  somewhere  in  the  middle  being  optimal). 
Too  little  concurrency  is  bad  for  a  program  as  serial  design  and  pro¬ 
duction  yields  a  longer  duration  (and  thus  more  cost)  before  fielding 
of  the  weapon.  Too  much  concurrency  is  also  bad  as  it  accepts  too 
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TABLE  1.  LIST  OF  WEAPON  SYSTEMS 


Program 

PNO 

AIM  9X  Sidewinder  Missile 

581 

Air  Warning  and  Control  System  Radar  System  Improvement 
Program  (AWACS  RSIP) 

524 

Bradley  Upgrade 

601 

C-17A  Globemaster  III 

200 

CH-47F  Improved  Cargo  Helicopter 

278 

EA-18G  Growler 

378 

F-22  Raptor 

265 

F/A-18  E/F  Super  Hornet 

549 

Family  of  Medium  Tactical  Vehicles  (FMTV) 

746 

High  Mobility  Artillery  Rocket  System  (HIMARS) 

367 

Joint  Air-to-Surface  Standoff  Missile  (JASSM) 

555 

Joint  Direct  Attack  Munition  (JDAM) 

503 

Joint  Primary  Aircraft  Training  System  (JPATS) 

560 

Longbow  Apache  Helicopter 

831 

Longbow  Hellfire  Missile 

541 

MH-60R  Seahawk®  Helicopter 

191 

MH-60S  Seahawk®  Helicopter 

282 

MHC  51  Osprey  Minehunter 

772 

Minuteman  III  Guidance  Replacement  Program  (MM  III  GRP) 

302 

Sense  and  Destroy  Armor  (SADARM) 

735 

Small  Diameter  Bomb  (SDB) 

354 

SFW 

275 

SSN-21  Seawolf-class  Attack  Submarine 

258 

SSN-774  Virginia-class  Attack  Submarine 

516 

Stryker  Light  Armored  Vehicle 

299 

T-45S 

240 

Tactical  Tomahawk  Missile 

289 

V- 22  Osprey  Tiltrotor  Aircraft 

212 

much  technical  risk.  Thus,  some  moderate  level  of  concurrency 
would  be  the  optimal  in  the  sense  that  it  minimizes  cost  growth. 
This  would  yield  a  curve  similar  to  that  shown  in  Figure  2. 

The  logic  behind  this  approach  for  planned  concurrency  is 
relatively  simple.  Program  managers  plan  for  a  certain  level  of 
funding  concurrency.  If  they  plan  for  too  much,  they  may  accept 
too  much  risk  that  could  yield  cost  growth.  On  the  other  hand,  too 


245 


Cost  Implications  of  Design/Build  Concurrency 


July  2011 


FIGURE  2.  HYPOTHETICAL  QUADRATIC  RELATION  BETWEEN 
COST  GROWTH  AND  CONCURRENCY 
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little  funding  concurrency  forces  them  to  create  completely  serial 
development/design  and  production  processes  that  prolong  pro¬ 
gram  duration  and  also  create  cost  growth.  In  sum,  the  planned  level 
of  concurrency  forces  managers  to  make  decisions  that  ultimately 
lead  to  cost  growth  if  either  too  much  or  too  little  concurrency  is 
accepted. 

The  logic  for  the  actual  concurrency  follows  along  similar  lines. 
Program  managers  may  or  may  not  have  planned  for  concurrency, 
but  events  led  to  the  situation  where  some  level  of  concurrency 
occurred,  which,  if  too  high  or  too  low,  led  to  excessive  cost  growth. 
Again,  the  assumption  is  that  some  intermediate  level  of  actual 
concurrency  would  be  the  optimum. 

In  all  cases,  this  simple  rule  can  be  specified  with  the  following 
function,  which  was  estimated  using  OLS: 

CostGrowth  =  b0  +  b^oncurrency  +  b Concurrency2  +  e  (1) 


Planned  Concurrency 

Our  first  model  explored  the  relation  between  planned  concur¬ 
rency  and  procurement  cost  growth.  The  results  are  reported  in 
Table  2. 

Observe  that  two  of  the  parameters  are  statistically  significant 
at  the  .10  level  (i.e.,  the  probability  that  the  parameters  are  less 
than  10  percent  is  zero),  and  the  fitted  line  does  give  us  a  U-shaped 
curve  (Figure  3).  However,  the  adjusted  R-squared  is  very  low,  which 
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TABLE  2.  OLS  RESULTS:  PROCUREMENT  COST  GROWTH  VS. 
PLANNED  CONCURRENCY 


Estimate 

Std  Error 

P-Value 

Intercept 

1.825 

0.484 

0.001 

Concurrency 

-5.014 

0.465 

0.052 

Concurrency2 

3.667 

2.273 

0.119 

Adjusted  R2 

0.137 

FIGURE  3.  FITTED  CURVE:  PROCUREMENT  COST  GROWTH  VS. 
PLANNED  CONCURRENCY 
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forces  us  to  conclude  that  the  quadratic  model  has  little  predictive 
power  of  procurement  cost  growth. 

Note  that  much  of  the  curvature  in  the  model  comes  from  one 
outlier.  To  see  how  well  the  model  improves  without  this  data-point, 
we  ran  the  same  model  excluding  the  outlier  (Table  3).  This  resulted 
in  no  improvement  to  the  model  at  all  and  slightly  less  curvature. 

Finally,  to  see  if  some  other  possible  relation  was  evident,  we  ran 
the  LOESS  smooth  curving  routine  on  all  of  the  data  including  the 
outlier.  We  then  bootstrapped  the  90  percent  interquartile  range 
to  see  how  well  conditioned  the  data  are  to  the  original  curve.  If 
the  data  are  from  a  common  model,  the  smoothed  curves  gener¬ 
ated  by  the  repeated  sampling  should  be  similar  to  the  original, 
and  the  confidence  intervals  defined  should  be  fairly  tight  around 
the  original  curve.2  The  results  of  these  exercises  using  the  outlier 
data-point  discussed  previously  and  excluding  this  data-point  can 
be  seen  in  the  figures  that  follow. 
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TABLE  3.  OLS  RESULTS:  PROCUREMENT  COST  GROWTH  VS. 
PLANNED  CONCURRENCY  (OUTLIER  EXCLUDED) 


Estimate 

Std  Error 

P-Value 

Intercept 

1.097 

0.390 

0.010 

Concurrency 

-1.907 

1.934 

0.334 

Concurrency2 

1.157 

1.764 

0.518 

Adjusted  R2 

0.021 

FIGURE  4.  PROCUREMENT  COST  GROWTH  VS.  PLANNED 
CONCURRENCY 
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As  we  can  see  from  Figure  4,  the  interval  using  the  outlier  data- 
point  is  extremely  wide.  For  example,  if  a  program  had  planned 
concurrency  of  .2,  then,  within  the  90  percent  interquantile  range, 
the  procurement  cost  growth  for  that  program  could  easily  range 
from  50  percent  to  over  100  percent. 

To  ensure  that  the  outlier  was  not  a  significant  factor  in  these 
results,  we  ran  the  same  experiment  excluding  this  data-point.  This 
did  not  improve  the  results  in  any  discernible  way  (Figure  5). 

In  spite  of  the  fact  that  the  confidence  intervals  around  the 
original  LOESS  curves  are  wide,  we  do  see  a  pattern  in  the  data 
that  suggests  that  low  levels  of  planned  concurrency  are  more 
problematic  than  higher  levels  of  concurrency.  Again,  turning  to 
the  data  without  the  outlier,  we  calculated  the  mean  cost  growth  in 
procurement  for  those  programs  with  planned  concurrency  levels 
under  30  percent  and  compared  it  to  the  means  for  those  programs 
with  planned  concurrency  over  30  percent.  Those  under  30  percent 
experienced,  on  average,  approximately  110  percent  cost  growth 
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FIGURE  5.  PROCUREMENT  COST  GROWTH  VS.  PLANNED 
CONCURRENCY  (OUTLIER  EXCLUDED) 
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while  those  over  30  percent  experienced  an  average  cost  growth  of 
approximately  50  percent.  This  difference  was  statistically  different 
at  the  95  percent  confidence  level. 

Actual  Concurrency 

We  next  turn  our  attention  to  procurement  growth  as  a  function 
of  actual  concurrency.  Table  4  shows  the  results  of  estimating  the 
quadratic  model  using  OLS.  As  in  the  case  with  planned  concur¬ 
rency,  only  the  intercept  parameter  /30  is  significant  at  the  .01  level. 
The  model  as  a  whole  has  an  adjusted  R-squared  of  -0.01889  indi¬ 
cating  that  the  model  has  little  explanatory  power.  Note  also  that 
the  fitted  line  in  Figure  6  is  concave,  which  is  the  exact  opposite  of 
what  our  hypothesis  was  (i.e.,  a  U-shaped  curve). 


TABLE  4.  OLS  RESULTS:  PROCUREMENT  COST  GROWTH  VS. 
ACTUAL  CONCURRENCY 


Estimate 

Std  Error 

P-Value 

Intercept 

1.037 

0.530 

0.062 

Concurrency 

-0.453 

2.399 

0.852 

Concurrency2 

-0.275 

2.168 

0.900 

Adjusted  R2 

-0.019 
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We  also  note  the  existence  of  an  outlier  that  could  exhibit  a  fairly 
large  effect  on  the  model  (Figure  6).  To  account  for  this  possibility, 
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FIGURE  6.  FITTED  CURVE:  PROCUREMENT  COST  GROWTH  VS. 
ACTUAL  CONCURRENCY 
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TABLE  5.  OLS  RESULTS:  PROCUREMENT  COST  GROWTH  VS. 
ACTUAL  CONCURRENCY  (OUTLIER  EXCLUDED) 


Estimate 

Std  Error 

P-Value 

Intercept 

0.762 

0.376 

0.054 

Concurrency 

-0.086 

1.684 

0.960 

Concurrency2 

-0.319 

1.521 

0.836 

Adjusted  R2 

-0.040 

we  ran  the  same  OLS  model  again  without  this  outlier.  The  results 
follow  (Table  5). 

Using  these  data,  the  model  still  performed  poorly  with  only  the 
intercept  being  significant  at  the  .10  level  (Figure  7).  Further,  the 
fitted  line  was  still  concave. 

Using  the  LOESS  smooth  curving  method,  we  examined  the  data 
to  see  if  other  relationships  could  possibly  explain  the  data  better 
than  a  simple  quadratic  function.  As  in  the  case  for  planned  con¬ 
currency,  the  confidence  interval  is  very  wide,  indicating  that  actual 
concurrency  is  also  a  poor  predictor  of  procurement  cost  growth. 

To  ensure  that  the  outlier  was  not  a  significant  factor  in  these 
results,  we  ran  the  same  experiment  excluding  this  data-point. 
This  did  not  improve  the  results  in  any  discernible  way  (Figures 
8  and  9). 

Again,  we  used  several  statistical  methods  to  discover  any  rela¬ 
tion  between  actual  concurrency  and  procurement  cost  growth. 
We  specifically  reject  the  notion  that  actual  concurrency  has  a 
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FIGURE  7.  FITTED  CURVE:  PROCUREMENT  COST  GROWTH  VS. 
ACTUAL  CONCURRENCY  (OUTLIER  EXCLUDED) 
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FIGURE  8.  CONFIDENCE  INTERVAL:  PROCUREMENT  COST 
GROWTH  VS.  ACTUAL  CONCURRENCY 
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quadratic  relation  to  procurement  cost  growth  and  find  no  other 
polynomial  relationship  that  was  consistent  with  the  data.  As  is  the 
case  of  planned  concurrency,  we  do  see  a  slight  dip  in  cost  growth 
for  those  programs  with  actual  concurrency  of  approximately  30 
percent  although  this  is  not  as  pronounced.  Thus,  our  conclusion  is 
that  actual  concurrency  of  RDT&E  and  production  funding  is  not  a 
strong  predictor  of  procurement  cost  growth  either. 
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FIGURE  9.  CONFIDENCE  INTERVAL:  PROCUREMENT  COST 
GROWTH  VS.  ACTUAL  CONCURRENCY  (OUTLIER  EXCLUDED) 
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Conclusions  for  Procurement  Cost  Growth 

In  all  cases,  we  reject  the  hypothesis  that  procurement  cost 
growth  is  related  to  any  measure  of  concurrency  in  a  way  described 
by  a  quadratic  function.  We  also  found  no  other  polynomial  rela¬ 
tion  that  strongly  supports  the  data.  While  using  the  LOESS  curve 
smoothing  routine  on  all  forms  of  concurrency  did  suggest  some 
other  possible  relation,  bootstrapped  confidence  intervals  indicate 
that  any  relation  between  the  two  is  very  weak.  Thus,  even  if  we 
accepted  the  implied  curvature,  the  predictive  power  of  the  model 
for  any  of  the  concurrency  measures  was  extremely  low.  In  sum,  we 
found  that  little  if  any  explanatory  power  of  concurrency  by  itself 
affects  procurement  cost  growth.  The  one  result  that  did  stand  out 
was  that  in  the  case  of  both  planned  and  actual  concurrency,  too 
little  concurrency  was  actually  more  problematic  than  too  much 
concurrency;  that  is,  concurrency  levels  under  approximately  30 
percent  were  associated  with  higher  average  levels  of  cost  growth 
and  higher  variance  as  well. 

Notably,  our  results  do  not  indicate  that  concurrency  is  never 
a  problem  for  programs  and  never  leads  to  cost  growth.  Rather, 
it  shows  that  concurrency  by  itself  is  insufficient  to  predict  cost 
growth.  Most  likely,  concurrency  leads  to  cost  growth  under  par¬ 
ticular  circumstances  or  in  the  presence  of  other  factors.  What  these 
circumstances  or  factors  are  is  not  clear  and  should  be  examined 
in  further  research. 
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ENDNOTES 

1.  Quoting  from  the  2009  “Greenbook”: 

DoD  arrives  at  the  figures  in  this  book  using  inflation  rates  published  by  the 
Office  of  Management  and  Budget  (OMB)  as  a  baseline.  OMB  typically  bases 
their  rates  on  Gross  Domestic  Product  (GDP)  composite  rates,  accounting  for 
non-pay  factors  only.  DoD,  however,  includes  pay,  fuel,  and  medical  accrual 
factors  in  its  composite  rates.  In  addition,  outlay  rates  are  factored  into  the  final 
DoD  inflation  rates.  (DoD,  2009) 

2.  The  LOESS  bootstrap  method  is  nonparametric,  implying  that  we  make  no  assumptions 
about  the  structure  of  the  error  term.  However,  we  measure  tightness  by  creating 

an  interval  around  the  original  curve  that  includes  90  percent  of  the  bootstrapped 
curves.  These  curves  approximate  the  5th  and  95th  percentiles  of  the  true  underlying 
distribution. 
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CHALLENGES  TO 
INNOVATION  IN 
THE  GOVERNMENT 
SPACE  SECTOR 


"  Zoe  Szajnfarber,  Matthew  G.  Richards, 
and  Annalisa  L.  Weigel 

This  article  uses  innovation  theory  to  identify  five  core  chal¬ 
lenges  of  generating  national  security  space  innovation:  (a) 
generating  bottom-up  push  in  a  top-down  environment;  (b) 
integrating  fragmented  buy-side  knowledge;  (c)  integrating 
fragmented  sell-side  knowledge;  (d)  matching  the  innovation 
environment  to  the  development  stage;  and  (e)  balancing  risk 
aversion  with  the  need  for  experimentation.  An  analysis  of  how 
the  current  two-tiered  process,  which  separates  technology 
development  from  project-based  acquisition,  addresses  these 
challenges,  reveals  that  this  method  of  separation  is  not  a 
complete  solution  because  it:  (a)  fails  to  value  architectural 
innovation;  (b)  creates  a  disaggregated  knowledge  base, 
which  exacerbates  the  difficulty  of  top-down  specification  and 
bottom-up  integration;  and  (c)  fails  to  generate  an  entrepre¬ 
neurial  supply-side  spirit.  Recommendations  for  improvement 
are  provided. 


Keywords:  Innovation  Theory ;  Monopsony-oligopoly 
Market,  Space  Acquisition,  Spacecraft  Development, 
Acquisition  Process,  Risk  Aversion 
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Characteristics  of  the  government  space  market,  with  its  mon¬ 
opsony-oligopoly  structure  and  complex  robust  products,  make 
encouraging  innovation  challenging.  The  Department  of  Defense 
(DoD)  acquisition  structure  represents  one  example  of  how  these 
challenges  are  addressed  in  an  institutional  setting.  However,  a 
recent  string  of  failures  has  brought  into  question  the  efficacy  of  the 
system.  Multiple  blue  ribbon  panels  have  been  convened  leading  to 
recommendations  about  how  the  current  system  can  be  improved; 
however,  these  recommendations  take  certain  implicit  assumptions 
of  the  system  as  a  given.  If  a  major  reform  is  to  be  achieved,  these 
fundamental  assumptions  must  be  reviewed.  This  article  takes  a 
step  back  from  the  acquisition  process,  using  innovation  theory 
to  assess  the  intrinsic  challenges  of  encouraging  complex  product 
innovation  in  a  government  monopsony-oligopoly.  In  particular,  it 
seeks  to  answer  the  following  questions:  (a)  What  are  the  implica¬ 
tions  of  the  space  sector  characteristics  on  innovation?  (b)  How  (or 
to  what  extent)  does  the  acquisition  system  address  these  implica¬ 
tions?  and  (c)  How  can  these  insights  be  used  to  improve  acquisition 
in  the  space  sector? 


Implications  of  Space  Sector 
Characteristics  for  Innovation 

Despite  a  rich  legacy  of  delivering  impressive  technology, 
defense  acquisitions  are  increasingly  characterized  by  schedule  slips 
and  cost  overruns.  With  long  development  times  and  high  complex¬ 
ity,  national  security  space  systems  (e.g.,  Advanced  Extremely  High 
Frequency  [AEHF],  National  Polar-orbiting  Operational  Environmen¬ 
tal  Satellite  System  [NPOESS],  Space-Based  Infrared  System-High 
[SBIRS-High],  Global  Positioning  System  [GPS]  II)  have  become 
particularly  illustrative  of  the  challenges  confronting  defense  acqui¬ 
sitions  (Government  Accountability  Office  [GAO],  2007).  In  recent 
years,  in  an  effort  to  address  these  problems,  multiple  blue  ribbon 
panels  have  been  convened.  Figure  1  enumerates  the  recommenda¬ 
tions  of  six  recent  reports  along  technical,  management,  and  policy 
dimensions. 

Bringing  to  bear  the  members’  vast  experience  working  in  the 
current  acquisition  paradigm  of  large  monolithic  spacecraft,  their 
recommendations  emphasize  a  back-to-basics  philosophy  (i.e., 
maturing  payload  technologies  outside  of  acquisition  programs). 
However,  with  the  rapidly  changing  requirements  that  characterize 
the  needs  of  today’s  warfighter,  it  may  be  the  acquisition  paradigm 
itself  that  needs  fixing. 
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FIGURE  1.  KEY  FINDINGS  FROM  RECENT  STUDIES 
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Note.  CINCSPACE  =  Commander  in  Chief,  Space  Command;  gov’t  =  government;  IOC  = 
Initial  Operating  Capability;  LSI  =  Lead  Systems  Integrator  ;  MDA  =  Milestone  Decision 
Authority;  NDIA  =  National  Defense  Industrial  Association;  NRC  =  National  Research 
Council;  NSC  =  National  Security  Council;  PM  =  Program  Manager;  R&D  =  Research  and 
Development;  req’s  =  requirements.  Adapted  from  DoD,  2006;  GAO,  2006;  GAO,  2007; 
NDIA,  2003;  NRC,  2008;  Rumsfeld  et  al.,  2001;  Young,  Hastings,  &  Schneider,  2003. 
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FIGURE  2.  OVERVIEW  OF  INNOVATION  CHALLENGES 
FOR  DEFENSE  ACQUISITIONS 
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By  applying  strategic  prescriptions  on  how  innovation  should 
be  encouraged  (as  abstracted  from  the  management  and  innova¬ 
tion  literatures)  to  intrinsic  characteristics  of  the  space  sector,  five 
fundamental  challenges  to  innovating  in  the  space  sector  were  iden¬ 
tified:  (a)  generating  bottom-up  push  in  a  predominantly  top-down 
acquisition  process;  (b)  representing  the  needs  of  a  disaggregated 
buyer;  (c)  integrating  fragmented  sell-side  knowledge  from  the 
top-down;  (d)  matching  the  innovation  environment  to  the  stage 
of  development;  and  (e)  balancing  risk  aversion  and  the  need  for 
experimentation.  Figure  2  provides  an  overview  of  these  five  chal¬ 
lenges.  The  following  sections  explain  the  nature  of  each  challenge. 

Challenge  1:  Generating  Bottom-Up  Push  in  a  Predominantly 
Top-Down  Acquisition  Process 

Taking  a  classical  economic  view  of  innovation,  market  trans¬ 
actions  are  thought  to  be  the  fundamental  driver  of  innovation. 
Innovation1  occurs  over  time  through  the  interaction  of  user  needs 
(market  pull)  and  seller  capabilities  (product  push)  (Rothwell  & 
Zegveld,  1994).  In  a  competitive  market,  this  process  happens 
naturally.  Both  the  consumer’s  willingness  to  pay  and  the  supplier’s 
ability  to  deliver  are  revealed  continuously  through  the  mechanism 
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FIGURE  3.  COMPARISON  OF  SPACE  AND  TRADITIONAL  MARKET 
STRUCTURES 
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of  price  (Adams  &  Adams,  1972).  However,  in  the  space  market, 
which  consists  of  only  one  buyer  and  few  sellers,  the  interaction 
only  occurs  when  the  monopsony  buyer  expresses  a  need.  As  a 
result,  the  transaction  is  less  effective  as  a  mechanism  for  revealing 
preference-capability  information. 

Figure  3  illustrates  the  differences  between  the  two  market 
types.  Since  monopsony  markets  are  discrete  (i.e.,  the  market  only 
exists  when  the  buyer  wants  to  buy),  buyer  needs  must  be  revealed 
explicitly  as  they  arise.  If  major  performance  improvements  are 
required  of  each  new  acquisition,  as  is  typically  the  case  between 
generations  of  spacecraft,  radical  innovation  must  occur  in  dis¬ 
crete  intervals,  upon  request.  Since  the  request  for  radical  change 
originates  from  the  government  buyer,  so  too  does  much  of  the 
investment  in  product  development  for  space  applications  (Sherwin 
&  Isenson,  1967).  The  market  is  dominated  by  a  top-down  “pull”  to 
the  near  exclusion  of  the  complementary  bottom-up  “push.”  This 
is  a  problem  because  a  fertile  innovation  environment  requires  the 
presence  of  both  forces,  especially  since  most  new  ideas  come  from 
outside  (Christensen,  2003).  Thus,  one  of  the  key  challenges  to 
innovating  in  the  space  market  is  for  the  government  to  encourage 
bottom-up  initiative. 
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Challenge  2:  Representing  the  Needs  of  A  Disaggregated  Buyer 

The  existence  of  a  top-down  acquisition  process  could  theo¬ 
retically  generate  ideal  conditions  for  innovation.  In  a  discrete 
and  specific  process  as  previously  described,  only  products  that 
the  buyer  wants  would  advance  to  the  development  stage.  This  is 
ideal,  assuming  the  buyer  knows  the  precise  product  specifications. 
However,  when  the  buyer  is  a  monopsonist  as  complex  as  the  U.S. 
Government,  incorporating  multiple  disaggregated  interests,  this 
assumption  may  be  invalid. 

As  illustrated  in  Figure  4,  in  the  government  acquisition  context 
the  monopsonist  buyer— which  encompasses  the  warfighter,  the 
appropriator,  acquirer,  and  taxpayer— is  not  a  single  coherent  deci¬ 
sion  maker.  The  monopsonist  exists  to  centralize  both  resources 
and  expertise.  As  a  result,  since  the  acquirers  (who  do  the  actual 
choosing)  must  integrate  operational  (warfighter’s  expertise)  and 
financial  (appropriator’s  knowledge  and  taxpayer  resources)  trades 
to  determine  what  next  to  buy,  the  decision  will  only  be  as  good  as 
their  imperfect  information.  Therefore,  unless  buyer  needs  are  well 
represented,  delivery  only  of  the  product  specified  in  the  request 
for  proposal  may  not  be  efficient  at  all. 


FIGURE  4.  OVERLAY  OF  KNOWLEDGE  AREAS  ON  SPACE 
ACQUISITION  MARKET  STRUCTURE 
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Challenge  3:  Integrating  Fragmented  Sell-Side  Knowledge 
from  the  Top-Down 

In  addition  to  knowing  what  it  wants,  efficiency  in  a  top-down 
acquisition  process  also  requires  that  the  buyer  knows  what  is  pos¬ 
sible.  In  a  commercial  setting,  typically  a  range  of  products  exists 
from  which  to  choose.  Even  when  buying  for  a  third  party,  a  his¬ 
tory  of  revealed  preferences  vis-a-vis  similar  products  provides  a 
reasonable  basis  upon  which  to  make  selections.  However,  in  the 
case  of  acquirers  buying  for  warfighters,  the  acquirers  have  only 
ever  seen  the  warfighters  use  other  systems  that  were  also  bought 
for  them.  While  this  intensifies  the  acquirers’  challenge,  it  also  pres¬ 
ents  a  unique  opportunity  for  the  monopsony  buyer  (as  a  whole) 
to  take  a  long-term,  coherent  perspective  on  driving  innovation  to 
their  benefit. 

However,  knowing  what  is  possible  is  particularly  difficult  in  the 
realm  of  complex  engineering  products  because  they  require  the 
integration  of  so  many  different  types  of  knowledge.  For  example, 
a  simple  communication  satellite  requires  the  technical  expertise 
of  thermal,  power,  solar,  control,  software,  structural,  and  electrical 
engineers  among  others.  In  the  time  between  successive  acquisi¬ 
tions  (often  10-20  years),  advances  will  have  likely  been  made  in 
each  discipline,  as  well  as  at  the  system  level.  In  order  to  manage 
this  complexity,  prime  contractors  whose  primary  expertise  is  sys¬ 
tems  integration  (i.e.,  architectural  knowledge  of  how  the  pieces 
fit  together)  have  emerged.  They  bid  for  whole  contracts  and 
farm  out  much  of  the  subsystem  development  effort  (component 
knowledge).  This  has  led  to  a  hierarchical  fragmentation  of  the 
knowledge  required  to  know  what  should  come  next  and  generate 
radical  innovation  (Henderson  &  Clark,  1990).  The  result,  as  shown 
in  Figure  4,  is  that  acquirers  are  not  in  a  strong  position  to  make  this 
determination;  sell-side  input  is  needed. 

Challenge  4:  Matching  the  Innovation  Environment  to  the 
Stage  of  Development 

Utterback  and  Abernathy  (Utterback,  1994;  Utterback  &  Aberna¬ 
thy,  1975)  have  shown  empirically  that  a  relationship  exists  between 
maturity  of  the  product  undergoing  innovation  and  characteristics 
of  the  organization  in  which  the  innovation  occurred.  Dividing 
the  innovation  process  into  three  phases— fluid,  transitional,  and 
specific— they  argue  that  free  experimentation  and  a  diversity  of 
ideas  are  important  ingredients  for  the  fluid  phase  (e.g.,  inventors 
working  out  of  their  garages),  while  increasingly  rigid  organiza¬ 
tional  processes  become  appropriate  as  the  product  matures  (e.g., 
a  promising  idea  gets  bought  out  and  commercialized  by  a  larger 
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firm).  The  differences  between  the  organizational  environments  are 
summarized  in  Table  1. 

Having  multiple,  different  innovation  environments  is  particularly 
important  for  space  systems  because  of  their  inherent  complex¬ 
ity.  Space  systems  decompose  into  subsystem  elements,  which 
decompose  into  component  elements,  etc.  At  each  level  of  inte¬ 
gration,  innovation  can  be  achieved  through  improvements  to  the 
element  itself,  or  the  way  in  which  it  interacts  with  other  elements. 
Both  types  of  innovation  are  required  to  achieve  radical  change,  as 
illustrated  in  Table  1  (Utterback,  1994).  Thus,  in  addition  to  the  fluid, 
transitional,  and  specific  phases  defined  by  Utterback  and  Aberna¬ 
thy,  spacecraft  development  may  require  additional  variants  to  deal 
with  both  the  component  and  architectural  dimensions  of  innova¬ 
tion2  (Sauser,  Ramirez-Marquez,  Magnaye,  &  Tan,  2008).  Yet,  since 
spacecraft  are  developed  as  a  single  project,  a  single  organizational 
environment  exists  throughout  the  formal  process.  As  a  result,  a  key 
challenge  involves  creating  an  organization  that  supports  multiple 
innovation  environments  simultaneously. 

Challenge  5:  Balancing  Risk  Aversion  and  the  Need  for 
Experimentation 

Perhaps  the  biggest  difference  among  the  three  phases  is  the 
extent  to  which  innovation  can  be  planned.  Once  a  dominant  design 
emerges  (in  the  transitional  phase),  innovation  can  be  achieved  by 
systematically  making  incremental  improvements  along  particular 
dimensions,  but  until  that  point,  there  is  much  less  certainty  about 
what  will  work.  In  the  transitional  and  specific  phase,  increasingly 
formal  organizational  structures  are  put  in  place,  and  those  struc¬ 
tures  facilitate  the  optimization  aspect  of  the  innovation  process. 
Conversely,  the  fluid  phase  start-ups  have  very  little  in  the  way  of 
organizational  structure,  in  part  because  no  consensus  has  yet 
emerged  on  how  the  creativity  is  best  encouraged  (Fagerberg, 
Mowery,  &  Nelson,  2005).  Another  reason  is  that  many  innovations 
fail  to  make  it  out  of  the  fluid  phase.  Most  successful  entrepreneurs 
failed  several  times  before  they  succeeded,  and  fail  again  many 
times  afterward.  These  are  not  risks  that  big  companies  typically 
take;  such  bold  risk  taking  requires  an  undying  belief  in  one’s  prod¬ 
uct  that  is  often  associated  with  entrepreneurs  (Casson,  Yeung, 
Basu,  &  Wadeson,  2006).  As  a  result,  society  does  not  have  a  high 
expectation  for  the  success  of  start-ups,  and  their  failure  is  not 
remarkable.  This  is  not  the  case  for  space  systems. 

Despite  the  fact  that  many  new  spacecraft  are,  for  all  intents 
and  purposes,  prototypes  (i.e.,  inventions)  at  the  system  level,  a 
high  level  of  risk  aversion  characterizes  the  U.S.  space  architecture. 
Many  reasons  are  cited  for  the  conservatism  that  exists  in  the  sys- 
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tern.  Unlike  most  terrestrial  systems,  once  a  spacecraft  is  launched, 
if  systems  fail  or  problems  arise,  fixing  them  is  extremely  difficult. 
Additionally,  the  act  of  launching  the  system,  which  is  the  only  way 
to  really  test  its  survivability  in  the  harsh  environment  of  space,  is 
extremely  expensive.  Thus,  an  extremely  high  premium  is  placed  on 
getting  it  right  the  first  time.  In  part  because  spacecraft  tend  to  be 
so  expensive,  failure  is  accompanied  by  a  high  political  cost.  Unlike 
in  the  fluid  phase  of  traditional  markets,  where  inventors  receive 
little  attention  until  they  succeed,  space  projects  are  highly  visible. 
What’s  more,  the  public  has  little  appreciation  for  the  experimental 
nature  of  most  first  flights,  reinforcing  the  need  to  succeed  the  first 
time.  However,  if  innovators  are  to  continue  surfacing  and  develop¬ 
ing  radically  different  solutions,  the  need  to  shelter  them  from  the 
constraining  pressures  of  success  becomes  an  imperative. 


DoD  Approach  to  Addressing  the  Challenges  of 
Spacecraft  Innovation 

Although  the  DoD  acquisition  framework  was  not  explicitly 
designed  to  address  the  five  challenges  previously  presented  herein, 
it  does  address  each  to  some  degree.  This  section  describes  the 
nature  of  the  interaction. 

Challenge  1:  The  Challenge  of  Generating  Bottom-up  Push  in  a 
Top-down  Structure  is  Addressed  Directly 

The  DoD  acquisition  process  employs  a  two-tiered  organiza¬ 
tional  structure  focused  on  (a)  research  and  development,  and  (b) 
formal  acquisition  programs.  Initial  technology  development  within 
the  DoD  is  conducted  by  the  Service  Laboratories  (e.g.,  Air  Force 
Research  Laboratory,  Naval  Research  Laboratory,  Army  Research 
Laboratory)  and  several  science  and  technology  organizations  such 
as  the  Air  Force  Office  of  Scientific  Research,  the  Office  of  Naval 
Research,  and  the  Defense  Advanced  Research  Projects  Agency. 
The  technology  development  tier  ensures  that  capabilities  that  will 
be  needed  in  the  future  are  under  development  today.  The  approach 
is  relatively  successful  in  generating  new  technologies,  but  is  limited 
in  two  important  ways.  First,  it  places  a  disproportionately  high  cost 
and  risk  burden  on  the  government  since  it  is  still  an  internal  organi¬ 
zation  writing  the  specifications.  Second,  a  manufactured  push  (as  is 
the  case  here)  is  not  the  same  as  a  true  bottom-up  push.  Where  the 
latter  embodies  the  results  of  multiple  organizations  competing  with 
each  other  to  find  the  best  solution,  the  former  remains  a  response 
to  a  request  for  progress  on  a  particular  technology. 
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Challenge  2:  The  Challenge  of  Representing  the  Needs  of 
a  Disaggregated  Buyer  is  Nominally  Addressed  Through 
the  Functions  of  the  Joint  Capabilities  and  Integration 
Development  System  (JCIDS)  Process 

JCIDS  constitutes  the  formal  DoD  procedure  for  the  estab¬ 
lishment  of  acquisition  requirements  and  evaluation  criteria  for 
future  defense  programs,  and  aims  to  assess  all  available  alterna¬ 
tives  for  meeting  a  validated  warfighting  need.  In  so  doing,  JCIDS 
seeks  to  integrate  the  preferences  of  multiple  stakeholders  in  the 
defense  establishment  by  examining  perceived  capability  shortfalls 
or  gaps  of  the  combatant  commanders  or  Secretary  of  Defense.  In 
theory,  JCIDS  should  address  the  challenge  identified  as  Challenge 
2  exactly;  but,  in  practice  the  complexity  of  integrating  the  needs 
of  such  a  disaggregated  buyer  as  the  U.S.  Government  leads  to 
significant  shortcomings  in  practice.  While  the  DoD  has  significant 
experience  translating  requirements  into  products,  the  department 
is  less  effective  at  flowing  needs  into  requirements— the  crux  of 
Challenge  2. 

Challenge  3:  The  Challenge  of  Integrating  Fragmented  Sell- 
side  Knowledge  has  Been  Addressed  Differently  Over  the 
History  of  the  Space  Age 

Initially,  significant  in-house  technical  expertise  was  cultivated 
among  government  buyers,  and  significant  oversight  spanning  the 
entire  sell-side  supply-chain  was  common  practice.  The  government 
buyer  adopted  the  risk  through  cost-plus  contracts  but  retained 
design  authority,  thus  giving  them  the  ability  to  intervene  when 
contracts  were  not  being  executed  as  desired.  More  recently,  as  cost 
control  became  a  primary  focus,  the  role  of  system  integrator  has 
been  delegated  to  industry  contractors,  with  technical  development 
subsequently  delegated  to  subcontractors.  The  idea  was  that  profit- 
maximizing  firms  will  allocate  resources  more  efficiently.  However, 
in  practice  the  interests  of  industry  do  not  always  align  with  those 
of  the  government,  limiting  the  effectiveness  of  the  relationship. 
Coupled  with  the  fact  that  the  delegation  of  the  oversight  role  has 
led  to  a  decrease  in  the  technical  competency  of  the  acquisition 
corps  (NRC,  2008),  this  trend  has  exacerbated  the  challenge  of 
integrating  sell-side  knowledge  rather  than  helped. 

Challenge  4:  The  Challenge  of  Matching  the  Innovation 
Environment  to  Stage  of  Development  is  Partially  Addressed 
by  the  Two-tiered  Acquisition  Structure,  in  that  Technology 
Development  is  Separated  from  Formal  Acquisition 

As  illustrated  in  Table  2,  this  separation  of  the  product  develop¬ 
ment  into  only  two  phases  makes  sense  if  technology  development 
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TABLE  2.  RELATIONSHIP  OF  ORGANIZATIONAL  STRUCTURE  TO 
PRODUCT  MATURITY 


Fluid 

Transitional 

Specific 

Innovation 

Characteristics 

Product 

changes/radical 

innovations 

Major  process 
changes, 

architectural 

innovation 

Incremental 

innovations, 
improvements  in 
quality 

Organizational 

Characteristics 

Entrepreneurial, 
organic  structure 

More  formal 

structure  with 

task  groups 

Traditional 

hierarchical 

organization 

Process 

Characteristics 

Flexible  and 

inefficient 

More  rigid  and 
changes  occur 
in  large  steps 

Efficient, 
capital-intensive, 
and  rigid 

Product  Maturity 

. > 

at  the  component  and  subsystem  levels  may  proceed  linearly  to 
spacecraft-level  system  integration.  However,  as  discussed  previ¬ 
ously,  TRL  is  only  one  component  of  product  maturity.  Product 
maturity  is  also  driven  by  architectural  knowledge  that  may  be 
measured  by  a  system’s  readiness  for  integration.  Developing 
new  technologies  for  components  and  subsystems  may  actu¬ 
ally  decrease  product  maturity  because  of  its  ability  to  modify 
architectural  knowledge  of  the  system.  Table  3  presents  a  more 
realistic  representation  of  the  evolution  of  product  maturity.  While 
formal  technology  development  processes  mature  technologies 
in  the  fluid  phase  up  to  the  subsystem  level  of  integration,  only  at 
the  spacecraft  level  is  integration  of  the  constituent  technologies 
addressed.  In  other  words,  the  formal  acquisition  process  (which 
has  the  organizational  characteristics  of  the  specific  phase)  is 
forced  to  develop  and  integrate  technologies  that  are  far  from 
specific  in  terms  of  maturity. 

Challenge  5:  The  Challenge  of  Balancing  Risk  Aversion  and  the 
Need  for  Experimentation  Faces  a  Similar  Partial  Fix 

While  the  technology  development  tier  serves  to  shelter  R&D 
and  component  maturation  from  the  public  eye,  no  such  shelter 
currently  exists  for  the  whole  space  system. 
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TABLE  3.  TECHNOLOGY  DEVELOPMENT  PATHS 

a.)  Theoretical  Two-Tiered  Progression 


Fluid 

Transitional 

Specific 

Components 

Technology 

Development 

Tier 

Subsystem 

Spacecraft 

'"•■j 

Formal 

Acquisition 

Maturity  . > 


b.)  Implementation  of  Two  Tiers 


Fluid  Transitional  Specific 


Components 

Technology 

Development 

Tier 

Subsystem 

Spacecraft 

L"' 

L' 

Formal 

. >  Acquisition 

Maturity  . 

. > 
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Guidelines  for  Improving  Innovation  in  the 
Spacecraft  Acquisition  System 

Over  the  last  decade,  multiple  blue-ribbon  panels  have  been 
convened  to  address  known  problems  with  the  acquisition  system. 
The  key  insights  from  these  reviews  are  summarized  in  Figure  1. 
Building  on  the  recommendations  therein,  this  section  highlights 
improvements  that  would  specifically  address  the  five  core  chal¬ 
lenges  to  space  sector  innovation,  as  identified  previously.  These 
recommendations  are  summarized  in  Table  3. 

From  an  innovation  theory  point  of  view,  in  Challenge  1— gen¬ 
erating  bottom-up  innovation— the  space  market  structure  inhibits 
half  of  the  natural  competitive  market  innovation  dynamic.  As  a 
result,  until  more  buyers  become  involved  in  the  space  market,3  any 
acquisition  system  will  need  a  mechanism  through  which  to  ensure 
that  new  ideas  continue  to  be  infused  into  the  acquisition  system. 
Development  contracts  do  accomplish  this  capability  development 
to  a  certain  extent,  but  as  discussed  previously,  they  are  limited  in 
their  ability  to  encourage  sell-side  initiative  and  the  parallel  and 
varied  concept  explorations  it  embodies.  Several  other  models 
exist  for  encouraging  and  leveraging  sell-side  initiative  including 
commercial  off-the-shelf,  seed-funding  models  being  explored  by 
the  Operationally  Responsive  Space  program  office  and  prizes  (e.g., 
Ansari  X-Prize).  The  idea  in  each  of  these  is  to  help  sustain  a  market 
rather  than  subsidize  the  development  of  a  particular  technology 
(i.e.,  generate  sell-side  initiative,  not  just  capability  development). 

With  regard  to  Challenge  2  (needs  representation)  and  Chal¬ 
lenge  3  (knowledge  integration),  the  blue  ribbon  panels  are  almost 
unanimous  in  their  recommendations  to  increase  the  technical 
competence  of  the  Defense  Acquisition  Workforce  and  empha¬ 
size  the  importance  of  front-end  specification.  However,  this  only 
addresses  half  of  the  problem.  No  matter  how  many  new  capabilities 
are  generated,  their  value  will  hinge  on  how  well  the  original  need 
was  represented  as  a  set  of  requirements.  For  the  other  half  of  the 
problem  to  be  fully  resolved,  more  emphasis  must  be  given  to  the 
challenge  of  knowledge  integration  on  both  the  buy-  and  sell-side. 
Specifically,  with  respect  to  Challenge  2,  increased  emphasis  must 
be  placed  on  flowing  needs  to  requirements.  This  will  involve  a  com¬ 
bined  effort  to  educate  users  about  their  choices  (what  is  possible) 
and  help  acquirers  capture  their  needs  more  effectively.  To  this 
end,  value-based  system  analysis  methodologies  to  facilitate  the 
process  of  capturing  both  articulated  and  unarticulated  needs,  early 
in  the  conceptual  design  phase,  are  currently  being  developed  by 
researchers.  Taking  the  value-centric  perspective  during  conceptual 
design  empowers  stakeholders  to  rigorously  evaluate  and  to  com- 
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pare  different  system  requirements  in  the  technical  domain  using 
a  unifying  set  of  attributes  in  the  value  domain  (Mathieu  &  Weigel, 
2005;  Ross,  Hastings,  Warmkessel,  &  Diller,  2004).  If  deployed  by 
system  program  offices,  these  emerging  system  analysis  method¬ 
ologies  will  contribute  significantly  to  overcoming  Challenge  2. 

Overcoming  Challenge  3  will  require  more  frequent  interactions 
among  contractors,  integrators,  and  the  government  through  for¬ 
mal  acquisitions.  Where  need-capability  information  is  transferred 
continuously  from  buyers  to  sellers  and  vice  versa,  in  traditional 
markets  the  transfer  only  happens  during  contracted  hardware 
development  in  the  space  sector.  As  long  as  space  acquisition 
continues  to  operate  on  a  model  of  infrequent,  extremely  complex 


monoliths,  the  knowledge  required  to  innovate  will  continue  to  be 
fragmented  across  the  various  players.  Decreasing  the  acquisition 
cycle  time  will  not  only  help  the  knowledge  integration  problem 
identified  in  Challenge  3,  but  also  the  risk  aversion  in  Challenge  5. 

Challenge  4  (matching)  identifies  a  fundamental  limitation  of  the 
current  system.  In  the  existing  acquisition  paradigm,  the  product 
development  required  to  enable  future  missions  is  conceptualized 
as  a  linear  progression  from  TRL  1-9.  With  this  view  in  mind,  the 
blue  ribbon  panels  call  for  increased  funding  for  technology  test¬ 
ing.  However,  while  increased  funding  for  technology  development 
is  a  needed  step  in  the  right  direction,  it  only  addresses  part  of  the 
problem.  It  fails  to  appreciate  the  difference  between  architectural 
and  component  dimensions  of  knowledge  and  what  that  means  for 
system-level  maturity.  If  the  rest  of  the  problem  is  to  be  addressed, 
a  need  arises  for  more  than  two  organizational  tiers:  one  for  each 
of  the  three  phases,  as  well  as  the  dimensions  of  component  and 
architectural  knowledge. 


272 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


Similarly,  the  recommendations  of  the  blue  ribbon  panels  that 
pertain  to  Challenge  5  (risk  shelter)  emphasize  a  back-to-basics 
philosophy,  which  keeps  R&D  separate  from  system  acquisition. 
This  would  serve  to  shelter  component  development  from  political 
pressures,  but  do  nothing  at  the  spacecraft  level.  For  spacecraft- 
level  development  to  receive  the  risk  shelter  that  is  required,  a  major 
philosophical  shift  is  needed.  In  this  case,  a  back-to-basics  philoso¬ 
phy  might  mean  a  return  to  the  CORONA  paradigm  (e.g.,  recall  that 
12  launches  of  the  revolutionary  CORONA  photoreconnaissance 
satellite  were  required  before  a  successful  demonstration  of  film  cap¬ 
sule  recovery  on  the  13th  flight  [Wheelon,  1995]).  In  other  words,  if 
radical  innovation  is  desired,  advanced  spacecraft  technology  must 
be  sheltered  from  the  ubiquitous  failure-is-not-an-option  mentality. 

The  challenges  identified  in  this  article  are  fundamental  to 
generating  innovation  in  the  space  sector;  they  will  not  be  easily 
overcome.  This  detailed  discussion  of  the  challenges  presented  in 
this  article  provides  some  guidelines  for  how  to  approach  solving 
their  associated  problems,  and  will  require  all  stakeholders  involved 
to  come  together  to  implement  a  solution. 
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ENDNOTES 

1.  For  purposes  of  this  article,  innovation  is  defined  as  a  measure  of  how  performance, 
normalized  by  resource  constraints,  changes  over  time.  This  can  involve  either  (a) 
generating  a  wholly  new  capability,  or  (b)  reducing  the  resources  required  to  achieve  an 
existing  capability  (e.g.,  making  the  system  cheaper  or  lighter). 

2.  While  component  innovation  is  achieved  through  technology  development  and 
measured  by  technology  readiness  levels  (TRL),  architectural  innovation  may  not  be 
explicitly  addressed  by  organizations.  To  support  the  formal  specification  of  product 
maturity  as  a  function  of  both  component  and  architectural  knowledge,  Sauser  et  al. 
(2008)  have  proposed  that  a  system  readiness  level  be  used  based  on  both  TRL  and  an 
integration  readiness  level  (IRL). 

3.  This  has  happened,  to  a  certain  extent,  in  the  domain  of  communication  satellites  and 
earth  imaging  and  may  soon  be  the  case  if  space  tourism  were  to  take  off,  but  is  arguably 
unrealistic  in  the  near  future  for  more  advanced  and  military  applications. 
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COST  GROWTH  IN  MAJOR 
DEFENSE  ACQUISITION: 

IS  THERE  A  PROBLEM? 
IS  THERE  A  SOLUTION? 

i  William  D.  O’Neil 

Cost  growth  in  defense  acquisition  is  both  a  problem  in  its  own 
right  and  part  of  the  larger  phenomenon  of  programs  that  fail 
to  perform  as  intended  or  desired.  It  is  a  limited  but  persistent 
phenomenon,  which  has  not  improved  in  any  material  respect 
over  at  least  the  past  four  decades;  nor  is  it  unique  to  defense, 
and  it  can  flow  from  a  variety  of  causes.  A  limited  group  of 
similar  remedies  have  repeatedly  been  tried,  but  achieved  very 
little  success  due  to  lack  of  clear  analysis  of  underlying  causes. 
Research  points  to  a  corrective  technique,  “taking  the  outside 
view,”  or  “reference  class  forecasting,”  with  clear  promise  for 
attacking  the  root  problems. 


Keywords:  Cost  Growth,  Program  Risk,  Cost  Sharing, 
Joint  Programs,  Cost  and  Analysis  Improvement 
Group  (CAIG) 


image  designed  by  Tia  Gray  » 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


The  reasons  for  concern  about  cost  growth  in  terms  of  its  influ¬ 
ence  on  Department  of  Defense  (DoD)  programs  were  succinctly 
reviewed  by  Mark  F.  Cancian  (2010).  In  this  article,  I  address  cost 
growth  in  defense  acquisition  both  as  a  problem  in  its  own  right 
and  as  a  part  of  the  larger  phenomenon  of  programs  that  fail  to 
perform  as  intended  or  desired.  I  show  in  turn  that:  (a)  it  is  a  limited 
but  persistent  phenomenon,  which  has  not  improved  in  any  material 
respect  over  at  least  the  past  four  decades;  (b)  it  is  not  unique  to 
defense;  (c)  cost  growth  may  flow  from  a  variety  of  causes— includ¬ 
ing  errors  in  the  management  or  contracting  process— but  defects  in 
the  original  concept  are  a  very  common  cause;  (d)  a  limited  group 
of  similar  remedies  have  repeatedly  been  tried  but  achieved  very 
little  success  due  to  lack  of  clear  analysis  of  underlying  causes; 
and  (e)  research  by  social  and  management  scientists  points  to  a 
corrective  technique,  “taking  the  outside  view”  or  “reference  class 
forecasting,”  which  has  a  sound  theoretical  basis  and  a  limited  but 
significant  record  of  success  in  nondefense  applications  as  well  as 
specific  defense  areas.  I  conclude  that  reference  class  forecasting 
and  its  supporting  analysis  and  data  collection  bases  should  be 
more  widely  adopted  in  defense  acquisition,  and  particularly  in  early 
evaluation  and  delineation  of  technical  issues. 


A  Limited,  But  Persistent  Problem 

In  the  United  States,  the  modern  era  of  concern  about  defense 
program  cost  and  results  can  fairly  be  said  to  have  started  in  the 
late  1960s  and  early  1970s.  Congress  began  demanding  Selected 
Acquisition  Reports  (SARs)  to  provide  much  better  and  more  com¬ 
prehensive  reporting  of  the  costs  of  Major  Defense  Acquisition 
Programs  (MDAPs)  (Cancian,  2010).  DoD  instituted  reforms,  includ¬ 
ing  establishment  of  the  Cost  Analysis  Improvement  Group  (CAIG) 
(Srull,  1998,  pp.  5-17),  presently  a  statutory  constituent  of  the  Cost 
and  Program  Evaluation  Office. 

Congress  has  repeatedly  revised  the  laws  governing  MDAPs, 
while  DoD  has  gone  through  more  than  a  dozen  substantively  differ¬ 
ent  generations  of  its  5000-series  acquisition  regulations  since  the 
first  versions  were  issued  in  July  1971  (Ferrara,  1996).  The  Obama 
Administration  followed  its  predecessors  in  instituting  a  spectrum 
of  reforms  and  initiatives  aimed  at  acquisition  improvement,  while 
one  of  the  incoming  president’s  early  acts  was  to  sign  into  law  a  new 
Weapon  Systems  Acquisition  Reform  Act  of  2009,  Pub.  L.  111-23. 
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The  Statistical  Record  of  Cost  Growth  in  DoD 

Examination  of  successive  annual  SARs  shows  that  when  signifi¬ 
cant  cost  growth  does  occur,  its  full  magnitude  rarely  is  apparent  for 
several  years  following  program  initiation,  and  frequently  not  for  10 
years  or  more— even  leaving  aside  growth  from  increased  ultimate 
production  quantities.  Thus,  it  will  be  years  before  the  real  results 
of  these  new  initiatives  can  be  objectively  assessed.  Indeed,  assess¬ 
ments  are  difficult  to  make  well  or  even  long  after  the  fact.  But  the 
best  and  most  comprehensive  assessment  of  MDAP  cost  growth  to 
date  has  concluded  that  up  through  programs  that  started  officially 
as  late  as  the  mid-1990s,  none  of  the  reforms  since  the  first  batch  in 
the  early  1970s  had  any  major  overall  effect  in  reducing  cost  growth. 
A  study  authored  by  Dr.  David  L.  McNicol  (2005,  pp.  18-19),  former 
chairman  of  the  CAIG,  now  with  the  Institute  for  Defense  Analyses 
(IDA),  deals  principally  with  procurement,  with  very  limited  detail  on 
development.  While  the  study  does  not  include  more  recent  results 
that  might  reflect  reforms  undertaken  early  in  the  2000s,  I  will  show 
that  limited  data  do  not  give  any  indications  of  improvement. 

This  is  only  one  of  a  number  of  analyses  that  attempt  to  deter¬ 
mine  trends  in  defense  acquisition  cost  growth.  Others  of  relatively 
recent  date  (Arena,  Leonard,  Murray,  &  Younossi,  2006;  Christensen, 
Searle,  &  Vickery,  1999;  Sipple,  White,  &  Greiner,  2004;  Smirnoff  & 
Hicks,  2008)  employ  various  statistical  techniques,  but  all  work  from 
the  historical  SAR  database  extending  back  to  December  1969,  with 
its  many  analytical  pitfalls.  Hough  (1992)  identified  the  most  notable 
problems  as:  (a)  failure  of  some  programs  to  use  a  consistent  base¬ 
line  cost  estimate,  (b)  exclusion  of  some  significant  elements  of  cost, 
(c)  exclusion  of  certain  classes  of  major  programs,  (d)  constantly 
changing  preparation  guidelines,  (e)  inconsistent  interpretation  of 
preparation  guidelines  across  programs,  (f)  unknown  and  variable 
funding  levels  for  program  risk,  (g)  cost  sharing  in  joint  programs, 
and  (h)  reporting  of  effects  of  cost  changes  rather  than  their  root 
causes. 

McNicol  (2005)  used  a  variety  of  approaches  to  avoid  or  miti¬ 
gate  the  effects  of  these  pitfalls.  He  started  with  data  refined  by 
adjusting  all  values  to  constant  2000  price  levels  and  constant 
quantities,  pruning  entries  not  really  relevant  to  rigorous  and  con¬ 
sistent  statistical  analysis  of  cost  growth,  employing  a  refined 
categorization  of  individual  cost  increases  to  distinguish  meaning¬ 
ful  trends,  and  further  adjusting  the  data  to  account  for  decisions 
to  change  requirements  or  budgets.  Then  he  used  the  standard 
econometric  technique  of  Ordinary  Least  Squares  Regression  analy¬ 
sis  of  panel  data  to  estimate  the  magnitude  and  significance  of  a 
wide  variety  of  causative  influences  on  cost  growth.  While  all  of  the 
analyses  agree  that  over  time  no  major  change  in  cost  growth  has 
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resulted  from  numerous  reform  efforts,  McNicol  (2005)  best  and 
most  rigorously  isolated  the  specifics;  accordingly,  I  largely  follow 
his  lead  in  analysis  of  causes. 

One  other  major  study  has  examined  the  SAR  data  with  similar 
care  to  provide  clear  insight  into  root  causes  (Bolten,  Lenonard, 
Arena,  Younossi,  &  Sollinger,  2008),  but  it  limits  its  scope  to  fewer 
than  one-third  as  many  programs  as  McNicol  (2005)  covered,  mak¬ 
ing  McNicol’s  the  superior  choice  for  purposes  of  this  study. 

After  pruning,  McNicol  was  left  with  138  MDAPs  that  passed  their 
Milestone  II  or  Milestone  B  (marking  formal  approval  as  programs 
and  entry  into  engineering  and  manufacturing  development  [EMD], 
and  approval  of  a  baseline  cost  estimate)  between  the  beginning 
of  1970  and  the  end  of  1997.  At  the  most  summary  level,  his  data 
are  plotted  in  the  Figure  as  the  solid  line  showing  the  distribution 
of  average  procurement  unit  cost  (APUC)  variance  from  baseline 
estimate.  While  few  programs  exactly  met  their  initial  procurement 
cost  estimates,  three-quarters  of  them  came  reasonably  close.  It 
is  the  smaller  number  of  very  high  growth  programs,  representing 
roughly  one  quarter  of  the  whole,  which  contributed  the  great  bulk 
of  overall  cost  growth. 

FIGURE.  PROCUREMENT  UNIT  COST  GROWTH  OF  MDAPs 
INITIALLY  APPROVED  BETWEEN  1970  AND  1997,  AND  PROGRAM 
UNIT  COST  GROWTH  OF  THOSE  APPROVED  BETWEEN  1998  AND 
2006 
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Extending  the  Statistical  Record 

McNicol’s  (2005,  p.  45)  data  did  not  extend  past  the  end  of 
1997.  Regrettably,  resources  to  update  the  data  set  he  currently 
uses,  which  would  permit  reanalysis,  have  not  been  forthcoming. 
Using  raw  gross  data  from  the  most  recent  SAR  summary  tables 
(DoD,  2009a),  however,  I  have  calculated  the  program  average  unit 
cost  (PAUC)  variances  for  those  programs  with  baselines  between 
1998  and  2006  and  plotted  the  distribution  of  these  as  the  series 
of  discrete  green  squares  in  the  Figure.  These  points  represent  only 
those  programs  that  had  their  initial  development  estimates  in  this 
period,  have  nonzero  procurement  quantity,  and  have  a  minimum 
of  3  years  EMD  since  the  initial  development  estimate— all  for  the 
greatest  possible  consistency  with  the  series  from  McNicol  (2005). 
The  most  notable  remaining  gross-level  inconsistency  is  that  the 
PAUC  data  include  development  and  military  construction  costs 
rather  than  solely  procurement  costs  as  detailed  in  McNicol  (2005), 
and  development  costs  on  the  whole  are  known  to  show  higher 
cost  growth  (McNicol  2005,  p.  17).  But  the  effect  of  this  is  mitigated 
because,  in  general,  procurement  cost  outweighs  development  by 
4:1  (McNicol  2005,  p.  4). 

Clearly,  the  two  distributions  plotted  in  the  Figure  show  the 
same  general  character,  with  that  for  the  more  recent  period  hav¬ 
ing  generally  higher  growth  in  the  upper  quartile.  A  two-sample, 
two-tailed  Kolmogorov-Smirnov  statistical  test  finds  inadequate 
evidence  to  accept  the  hypothesis  that  the  1998-2006  sample  is 
drawn  from  the  distribution  of  the  1970-1997  sample  even  at  the  80 
percent  significance  level  (p  =  0.638).  Because  of  the  differences 
in  the  two  data  sets,  we  must  not  read  too  much  into  this  result, 
but  clearly  the  statistical  test  reveals  no  evidence  that  even  hints 
of  secular  improvement  in  control  of  cost  growth,  at  least  through 
2006. 

In  both  the  earlier  and  later  samples,  we  see  that  roughly  three- 
quarters  of  the  included  MDAPs  have  reasonably  satisfactory  cost 
growth  histories,  with  at  most  no  more  than  30  percent  growth 
and  average  growth  near  zero.  Excessive  cost  growth  affects  only 
a  minority  of  programs. 

To  obtain  a  statistically  consistent  sample,  the  results  shown 
in  the  Figure  put  aside  programs  that  are  terminated  early,  that 
are  radically  restructured,  or  that  follow  significantly  nonstandard 
development  paths.  Recent  examples  include  the  Army  Future 
Combat  System  and  Navy  Littoral  Combat  Ship.  Such  programs 
often  have  high  cost  growth  and  thus  cannot  be  neglected  in  con¬ 
sidering  effects  and  cures,  but  their  omission  does  little  to  affect  the 
overall  statistical  picture.  Some  of  the  1990s-era  programs  shown 
could  well  experience  further  cost  growth,  since  the  most  seriously 
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troubled  programs  tend  to  involve  considerable  extensions  in  devel¬ 
opment.  Two  notable  examples  are  the  4-year  slip  in  the  schedule 
for  completion  of  the  F-35  Joint  Strike  Fighter  EMD  (DoD,  2010)  and 
approximately  9-year  slip  in  schedule  for  the  Space  Based  Infrared 
Satellite  (SBIRS)  High  (DoD,  2008). 

Public  discussions  of  defense  cost  growth  often  make  it  seem 
like  a  problem  unique  to  DoD,  but  this  gives  a  distorted  impression 
that  impedes  accurate  understanding  and  effective  correction. 
In  fact,  complex  programs  throughout  government  and  private 
industry  are  very  prone  to  cost  growth  (Flyvbjerg,  Bruzelius,  & 
Rothengatter,  2003;  Lovallo  &  Kahneman,  2003;  Merrow,  Phillips, 
&  Myers,  1981;  New  York  Times,  2011). 

The  Futility  of  Relying  on  Price  Competition 

Every  incoming  DoD  administration  has  made  efforts  to  improve 
the  management  of  acquisition,  with  control  of  cost  growth  usually 
a  prominent  declared  objective.  But  to  a  very  great  extent,  lack  of 
accurate  diagnosis  of  causes  has  undermined  these  initiatives.  Nota¬ 
bly,  a  review  of  a  pair  of  foundational  studies  of  defense  acquisition 
performed  half  a  century  ago  by  Merton  J.  Peck  and  Frederic  M. 
Scherer  of  the  Harvard  Business  School  reveals  significant  issues 
still  largely  unaddressed  by  intervening  management  efforts  (Peck 
&  Scherer,  1962;  Scherer,  1964).  In  particular,  Peck  and  Scherer 
(1962)  argued  at  length  that  price  competition— a  wide  favorite  for 
controlling  costs— is  bound  to  be  largely  ineffective  in  major  defense 
system  acquisition,  and  very  likely  counterproductive. 

Nevertheless,  officials  have  repeatedly  emphasized  price  com¬ 
petition  in  acquisition.  They  have  advocated  price  competition 
under  a  variety  of  banners,  with  the  common  element  being  an 
attempt  to  include  a  firm  commitment  regarding  production  of 
at  least  the  initial  lots  as  an  important  element  in  selecting  the 
development  contractor,  thus  transferring  the  risk  of  cost  growth 
to  the  contractor. 

In  principle  this  seems  sound  and  businesslike.  Cases  exist 
where  it  has  seemed  to  work  reasonably  well,  but  only  in  limited 
circumstances.  The  six  cases  of  this  approach  that  were  covered  in 
McNicol  (2005)  all  had  especially  high  cost  growth,  putting  them  in 
the  upper  quartile,  as  shown  in  the  Figure.  A  more  recent  example  is 
the  SBIRS  High,  which  attempted  a  modified  version  of  this  strategy. 
SBIRS  High  has  suffered  especially  great  cost  growth  (DoD,  2010), 
with  more  than  175  percent  reported. 

Attempts  to  transfer  the  risks  of  cost  growth  to  the  contrac¬ 
tors  fail  in  much  the  same  way  that  the  nation’s  banking  system 
collapsed  in  2008,  and  for  broadly  parallel  reasons.  Even  though 
the  remaining  major  defense  contractors  are  at  little  risk  of  being 
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allowed  to  go  out  of  business,  the  fortunes  of  their  individual  busi¬ 
ness  units  can  fluctuate  a  great  deal.  Their  managers  can  and  fairly 
frequently  do  suffer  diminished  career  prospects  and  even  job  loss 
when  things  go  wrong— a  powerful  negative  motivation.  But  they 
face  a  painful  dilemma.  If  they  promise  too  much,  then  they  may 
come  to  regret  it  in  a  few  years.  Yet,  if  they  promise  too  little,  they 
will  lose  out  at  once  to  a  competitor.  In  such  circumstances,  the 
incentives  weigh  heavily  on  the  side  of  accepting  future  risks  rather 
than  immediate  ones,  for  one  can  always  hope  for  some  redemptive 
development  in  the  meantime. 

The  critical  faculties  of  the  corporate  leaders  who  must  ulti¬ 
mately  approve  the  offer  are  blunted  by  the  knowledge  that  they 
command  an  organization  too  big  and  vital  to  be  allowed  to  fail.  A 
program  filled  with  problems  may  cause  pain,  but  not  corporate 
destruction.  And  like  their  subordinate  business  unit  managers,  they 
may  well  hope  for  some  future  deliverance. 

No  plausible  threats  of  retribution  for  distant  problems,  however 
dire,  can  go  far  to  offset  these  mechanisms.  In  principle  the  govern¬ 
ment  can  reject  offers  deemed  unrealistic,  as  it  does  when  offerors 
omit  some  significant  element  or  make  a  demonstrable  error.  But 
a  source  selection  authority  (SSA)  cannot  simply  substitute  his 
or  her  own  judgment  for  the  contractor’s  regarding  prospective 
improvement  or  advances  in  development  or  production.  Even  at 
best,  attempting  to  distinguish  degrees  of  realism  among  compet¬ 
ing  proposals,  in  many  cases,  is  fraught  with  unforeseen  difficulties. 

If  the  contractor  is  to  be  held  responsible,  the  government  must 
allow  it  much  autonomy  and  authority.  In  programs  where  price 
competition  is  not  central,  the  government  may  step  in  and  provide 
essential  assistance  and  direction  when  a  contractor  encounters  dif¬ 
ficult  problems,  but  this  is  inconsistent  with  holding  the  contractor 
responsible.  Individual  case  studies  of  such  programs  often  show 
contractors  running  into  trouble  while  responsible  officials  hesitate 
to  intervene.  Most  detailed  case  studies  contain  sensitive  informa¬ 
tion  and  remain  unpublished,  but  this  effect  can  be  clearly  seen  in 
Whittle  (2010)  and  Younossi  et  al.  (2008). 

Other  Inadequate  Explanations  and  Solutions 

Sometimes  problems  may  be  solved,  or  at  least  improved, 
without  thoroughly  analyzing  their  causes.  After  four  decades  of 
failed  attempts,  however,  we  have  to  question  how  long  it  might 
take  to  make  much  progress  against  cost  growth  and  its  companion 
problems  through  cut-and-try. 
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Some  usual  suspects  can  be  dismissed  from  the  lineup  at  once 
on  the  basis  of  strong  alibis.  These  include: 

Profiteering.  Defense  contractors  are  not  noted  for  high  profit  rates, 
and  executive  compensation  is  not  a  major  expense  in  this  industry. 

From  the  government’s  perspective,  the  function  of  profits  is  to 
permit  industry  to  raise  the  capital  it  needs  to  serve  government 
needs.  Contracting  policy  is  shaped  in  various  ways  to  minimize  the 
levels  of  profits  necessary,  and  analysis  shows  that  in  general  this 
is  achieved  efficiently.  Profitability  could  not  be  significantly  lower 
without  impairing  industry’s  ability  to  meet  government  needs 
(Arnold,  2008,  pp.  13-15). 

Lack  of  incentives  to  economize  or  reduce  inefficiency.  Throughout  the 
history  of  American  defense  contracting,  concerns  have  repeatedly 
been  expressed  that  in  the  absence  of  immediate  and  direct 
competitive  pressures  at  every  stage,  firms  would  lack  incentives 
to  economize  (Holley,  1964).  Close  analysis  by  Arnold,  McNicol,  and 
Fasana  (2009),  however,  showed  that  on  the  whole,  government 
contracting  officers  make  quite  effective  use  of  legally  permitted 
contract  incentives  to  motivate  performance. 

Experience  in  working  within  or  close  to  defense  industry  firms 
and  government  acquisition  organizations  reveals  many  areas  of 
apparent  inefficiency  or  waste— ill-motivated  or  poorly  qualified 
personnel,  idle  resources,  deteriorated  equipment,  bureaucratic 
busywork,  minor  peculation,  and  a  host  of  others.  Yet  on  the  whole, 
the  experience  is  not  noticeably  different  in  nondefense  industry. 
Where  it  has  been  possible  to  make  more  or  less  direct  compari¬ 
sons,  they  have  revealed  no  systematic  deficiency  in  defense-related 
efficiency  (Besselman,  Arora,  &  Larkey,  2000;  Kelley  &  Watkins, 
1998).  The  pattern  in  which  a  relatively  small  proportion  of  programs 
account  for  virtually  all  of  MDAP  cost  growth  cannot  be  explained 
by  industry  inefficiencies  unless  they  are  somehow  specific  to  par¬ 
ticular  programs. 

Requirements  creep,  Requirements  changes  do  occur  and  they 
contribute  to  cost  growth.  But  the  cost  data  set  used  by  McNicol 
(2005)  adjusted  for  requirements  changes;  thus,  they  did  not 
contribute  to  the  pattern  of  cost  growth  seen  in  the  Figure. 

Technology  risk,  Another  usual  suspect  is  in  fact  more  commonly 
implicated  in  major  cost  growth:  excessive  technology  risk.  Public 
Law  111-84  (Armed  Forces,  2009)  requires  certification  at  the  time 
of  program  initiation  that  “the  technology  in  the  program  has  been 
demonstrated  in  a  relevant  environment.”  This  corresponds  to 
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what  the  Department  of  Defense  (2009b)  defined  as  Technology 
Readiness  Level  Six  (TRL  6).  The  Government  Accountability 
Office  (GAO),  in  its  periodic  assessments,  regularly  emphasizes 
technology  readiness,  which  it  cites  as  a  major  factor  in  determining 
the  prevalence  and  seriousness  of  cost  growth.  Levels  of  technology 
maturity  at  program  initiation  have  been  rising  in  recent  years, 
which  the  GAO  sees  as  an  encouraging  sign  for  future  control  of 
cost  growth  (GAO,  2009,  pp.  16-17). 

But  cost  growth  is  by  no  means  consistently  a  result  of  low  tech¬ 
nology  maturity.  The  Expeditionary  Fighting  Vehicle  (EFV)  program 
is  a  notable  example.  More  than  a  quarter  of  a  century  of  focused 
technology  development  efforts  preceded  program  approval  in 
2000,  including  the  construction  of  a  series  of  functional  prototype 
vehicles.  All  but  one  of  the  program’s  critical  technologies  met  TRL 
6,  and  the  remaining  one  has  not  caused  prohibitively  expensive 
problems.  Nevertheless,  the  engineering  prototypes  functioned  so 
badly  that  testing  had  to  be  abandoned,  and  EMD  had  to  be  started 
over  again.  Planned  procurement  has  been  cut  more  than  43  per¬ 
cent,  objectives  for  performance  and  reliability  have  been  scaled 
back  substantially,  scheduled  initial  operational  capability  has  been 
slipped  by  approximately  9  years,  and  the  estimate  of  APUC  has 
risen  by  168  percent  (DoD,  2008). 

In  the  EFV  as  in  many  other  high-growth  programs,  the  fun¬ 
damental  problem  is  not  technology  per  se  but  failure  to  work  out 
and  recognize  in  advance  many  of  the  implications  of  the  design 
choices  that  were  made  at  the  time  of  program  initiation.  We  can 
trace  a  high  proportion  of  the  problems  in  the  current  and  former 
“leaders”  in  cost  growth  to  variations  on  this  theme.  Program  man¬ 
agers  and  engineers  laid  confident  plans  to  achieve  performance 
and  schedule  goals  without  recognizing  what  they  truly  involved. 
This  can  be  clearly  seen  in  a  few  published  program  case  studies 
(Coulam,  1977;  Whittle,  2010;  &  Younossi  et  al.,  2008),  but  other 
studies  remain  unpublished  due  to  sensitivity. 

The  Origins  of  Flawed  Plans 

How  can  this  be?  How  can  experienced  and  well-qualified  man¬ 
agers  and  engineers  repeatedly  fail  to  lay  realistic  plans?  How  can 
acquisition  officials  repeatedly  overlook  such  faults,  often  bending 
or  setting  aside  established  policies  to  do  so?  Modern  research  in 
social  and  management  sciences  provides  answers,  involving  pat¬ 
terns  of  behavior  at  both  the  individual  and  group  level. 

At  the  individual  level,  the  key  factor  is  the  planning  fallacy. 
This  is  a  concept  growing  out  of  the  work  of  Daniel  Kahneman 
and  Amos  Tversky  (1977, 1982).  The  phrase  refers  to  the  pervasive 
human  tendency  to  hold  “the  conviction  that  a  current  project  will 


286 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


go  as  well  as  planned  even  though  most  projects  from  a  relevant 
comparison  set  have  failed  to  fulfill  their  planned  outcomes.”  Con¬ 
trolled  experiments  have  repeatedly  validated  the  phenomenon 
(Buehler,  Griffin,  &  Peetz,  2010). 

Management  and  social  scientists  have  explored  the  planning  fal¬ 
lacy’s  operations  and  implications  specifically  in  business  (Lovallo  & 
Kahneman,  2003;  Flyvbjerg,  Lovallo,  &  Kahneman,  2003)  and  major 
infrastructure  projects  (Flyvbjerg,  Garbuio,  &  Lovallo,  2009).  Many 
of  the  problems  they  found  in  particular  cases  traced  to  faulty  deci¬ 
sions  related  to  the  planning  fallacy  (Buehler,  Griffin,  &  Peetz,  2010). 

At  the  group  level,  the  scenario  these  studies  present  as  typi¬ 
cal  involves  individuals  and  groups  competing  to  secure  adoption 
of  their  proposals  for  a  new  program.  They  are  driven  to  making 
unrealistic  promises,  in  exactly  the  same  manner— as  I  have  already 
argued— indicative  of  firms  competing  for  contracts.  That  is,  the 
groups  that  make  the  most  optimistic  promises  gain  an  advantage, 
so  long  as  their  optimism  does  not  excite  outright  incredulity.  Their 
optimism  is  fostered  by  their  own  planning  fallacies,  and  once  deci¬ 
sion  makers  have  bought  into  a  proposed  program,  they  too  are 
drawn  into  planning  fallacy. 

Explicit  strategic  deception  may  possibly  be  involved  at  one 
level  or  another,  deliberately  calculated  to  gain  advantage  over 
competing  proposals  (Flyvbjerg,  Holm,  &  Buhl,  2002;  LaBerge, 
1982).  But  very  unrealistic  plans  can  come  into  being  and  gain 
approval  without  Machiavellian  calculation,  particularly  in  a  cascade 
of  multiple  levels  of  decision  with  associated  multiple  layers  of  plan¬ 
ning  fallacy.  In  defense  acquisition,  my  experience  suggests  that  this 
is  far  more  common  than  calculated  deception. 

The  planning  fallacy  appears  to  be  a  given  fact  of  the  innate 
workings  of  human  thought.  It  is  extremely  difficult  to  see  it  in 
ourselves,  and  the  practically  minded  people  who  predominate  in 
decisions  regarding  acquisition  programs  seem  particularly  resistant 
to  such  introspection.  But  we  can  see  it  outside  of  ourselves,  if  we  are 
able  to  look  dispassionately,  and  that  offers  an  important  clue  about 
what  might  be  done  to  mitigate  its  ill  effects  (Buehler  et  al.,  2010). 

Kahneman  and  his  colleagues  suggest  what  they  call  taking  the 
outside  view,  or  reference  class  forecasting,  founded  in  a  process 
of  analyzing  data  from  the  results  of  prior  programs  or  efforts  that 
correspond  closely— as  closely  as  possible— to  what  is  planned.  Even 
though  the  correspondence  is  not  exact,  this  procedure  provides  a 
more  reliable  guide  to  results  than  forecasting  directly  on  the  basis 
of  detailed  program  plans  (Flyvbjerg,  2008). 


287 


Cost  Growth  in  Major  Defense  Acquisition: 
Is  There  a  Problem?  Is  There  a  Solution? 


July  2011 


The  Role  of  the  CAIG 

This  sounds  much  like  what  the  CAIG  has  been  doing  in  a  sophis¬ 
ticated  and  rigorous  way  for  the  past  four  decades,  using  what  it 
terms  the  parametric  method.  As  the  CAIG’s  first  director  described 
it,  “The  parametric  approach  does  not  rely  on  a  detailed  description 
of  the  ‘inputs’  to  the  system,  but  rather  considers  system  ‘output’ 
characteristics  such  as  speed,  thrust,  etc.  Historical  defense  sys¬ 
tem  cost  experience  is  used  to  develop  relationships  between  such 
output  characteristics  and  system  costs.  These  empirical  relation¬ 
ships  are  then  used  to  project  a  portion  or  all  of  the  costs  of  a  new 
system”  (Srull,  1972). 

In  some  cases,  the  CAIG  may  make  early  estimates  using  analo¬ 
gies  with  generally  similar  systems,  but  there  too  it  seeks  rigor 
through  the  use  of  structured  and  objectively  evaluated  selection 
of  analogues.  In  either  event,  it  is  pursuing  the  “outside  view,”  as 
Tversky  and  Kahneman,  and  those  who  have  followed  them,  have 
recommended  (Buehler  et  al .,  2010;  Flyvbjerg,  2008). 

Establishment  of  the  CAIG  was  followed  by  a  large,  swift 
improvement  in  agreement  between  official  estimates  and  actual 
costs,  even  though  acquisition  officials  were  not  required  to  accept 
its  recommendations  and  only  rarely  did  in  full  (McNicol,  2005). 
Viewed  from  outside  the  CAIG,  it  seemed  clear  to  me  at  the  time 
that  the  knowledge  that  estimates  were  reviewed  led  to  increased 
attention  to  cost  estimating  by  program  managers  and  the  spon¬ 
soring  organizations,  and  increased  willingness  to  adopt  the  CAIG’s 
parametric  methods.  This  was  fostered  by  its  active  efforts  to 
share  its  data  and  methods.  Thus,  the  CAIG  brought  a  measure  of 
cooperative-competitive  synergy  to  cost  estimation. 

Effective  Measures 

Impressive  as  it  is,  the  record  of  the  CAIG  (and  of  its  methods  in 
other  hands)  has  limitations.  DoD  treats  CAIG  estimates  as  sensitive 
management  information  and  does  not  release  them,  but  based  on 
seeing  many  over  the  past  four  decades,  it  seems  clear  to  me  that 
they  are  usually  more  accurate  than  (and  higher  than)  the  Service 
estimates,  but  also  sometimes  significantly  inaccurate.  Unpublished 
case  histories  of  some  high-growth  programs  show  costs  growing 
well  beyond  even  CAIG  forecasts.  Even  a  very  intensive  examination 
with  full  access  has  failed  to  find  enough  relevant  data  to  permit  a 
comprehensive  statistical  analysis  of  the  CAIG’s  historical  accuracy, 
but  does  make  it  clear  that  there  are  incidents  of  substantial  under¬ 
estimation  (McNicol,  Tyson,  Hiller,  Cloud,  &  Minix,  2006). 
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Its  authors  remark: 

The  estimates  prepared  by  cost  estimators  are  crucially 
dependent  on  technical  and  programmatic  assumptions 
over  which  they  have  little  or  no  say.  There  are  some  gray 
areas;  cost  estimators  should  recognize— and  provide  cor¬ 
rections  for  in  their  estimates— some  types  of  unrealistic 
program  assumptions  and  some  likely  execution  problems. 
But,  without  trying  to  fix  the  boundaries  of  these  exceptions, 
it  is  clear  that  they  are  exceptions— cost  estimators  generally 
are  not  equipped  to  do  engineering  analyses  of  proposed 
programs  or  to  assess  the  capabilities  of  potential  contrac¬ 
tors.  (McNicol,  2005,  p.  19) 

The  unpublished  case  studies  suggest  this  as  a  very  significant 
cause  of  serious  underestimates.  In  most  of  these  cases,  it  was  pos¬ 
sible  to  know  that  the  technical  assumptions  were  optimistic,  and 
this  was  pointed  out  by  at  least  some  observers  at  the  time.  While 
no  comprehensive  survey  has  been  conducted,  in  confidential  inter¬ 
views  CAIG  personnel  have  told  me  that  in  some  cases  they  had 
reservations,  but  ultimately  lacked  a  strong  basis  for  questioning 
confident  assertions  by  program  managers  or  other  official  advo¬ 
cates.  Thus,  while  no  basis  exists  for  assessing  the  incidence  of  such 
situations,  we  can  be  sure  it  is  not  zero. 

This  relates  to  what  then-Deputy  Secretary  of  Defense  David 
Packard  (1970)  emphasized  four  decades  ago:  Cost  growth  is 
closely  related  to  technical  problems  including  schedule  slippage, 
quality  problems,  and  inability  to  meet  baseline  requirements  (Fly- 
vbjerg  et  al.,  2003).  Faulty  initial  engineering  plans  and  concepts 
are  not  the  root  of  all  cost  growth,  but  are  involved  in  much  of  it. 

These  problems  can  be  attacked  by  an  approach  comparable 
to  that  which  the  CAIG  uses— taking  an  outside  view,  using  refer¬ 
ence  class  forecasting  of  technical  factors  as  well  as  the  costs  that 
depend  on  them.  The  basic  techniques  for  parametric  analysis  of 
engineering  characteristics  are  well  established  and  have  been 
used  by  engineers  for  at  least  250  years  in  the  early  design  phases 
of  systems  of  many  kinds  (Vincenti,  1990,  pp.  138-141).  They  are  a 
great  deal  like  the  techniques  used  by  the  CAIG  in  that  they  do  not 
depend  on  highly  detailed  information  about  the  system  design  or 
particular  technologies.  Those  who  apply  them  must  have  appro¬ 
priate  broad  technical  knowledge  and  judgment,  but  do  not  need 
deep  expertise  in  the  particular  systems. 

When  one  examines  program  development  histories  closely,  as 
I  often  have,  it  becomes  apparent  that  there  are  cases  in  which  the 
problems  were  such  that  even  thorough  engineering  parametric 
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analysis  might  not  have  identified  them,  but  many  more  in  which  it 
should  have— if  it  were  tried.  Unless  and  until  it  is  tried  in  a  system¬ 
atic  way  by  competent  personnel,  there  will  be  no  way  to  be  sure. 
But  given  the  historical  evidence  regarding  the  value  of  engineering 
parametric  analysis  generally,  together  with  the  modern  evidence 
regarding  the  importance  of  the  “outside  view,”  it  seems  that  a 
thorough  trial  is  called  for— and  all  the  more  so  since  the  cost  of 
such  efforts  is  so  small  compared  to  the  costs  of  even  one  badly 
conceived  or  executed  program. 
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CREATING  AND 
SUSTAINING 
AN  EFFECTIVE 
GOVERNMENT-DEFENSE 
INDUSTRY  PARTNERSHIP 

f  Steve  Mills,  Scott  Fouse,  and  Allen  Green 

U.S.  history  is  replete  with  enterprises  that  succeeded  due 
to  effective  partnerships.  Today,  the  nation’s  most  complex 
partnership  is  the  joint  pursuit  of  the  world’s  best  combat 
capabilities  by  the  U.S.  Department  of  Defense  and  the 
defense  industry.  These  two  complex  enterprises,  on  behalf 
of  the  nation  and  its  allies,  are  actively  developing,  producing, 
fielding,  and  sustaining  combat  systems  for  joint  warfighters 
that  are  second  to  none.  Does  this  shared  interaction  form 
an  effective  partnership?  In  this  article,  the  authors  analyze 
private  industry’s  perception  of  the  challenges/opportunities 
that  exist  in  the  shared  relationships  with  their  government 
counterparts.  Their  findings  pinpoint  five  focus  areas,  with 
corresponding  actions,  which  can  improve  the  partnership 
between  government  and  the  defense  industry. 


Keywords:  Partnership,  industry,  Program  Manager, 
Request  for  Proposal  (RFP),  Request  for  information 
(RFI) 
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In  1803,  President  Thomas  Jefferson  was  diligently  searching 
for  the  best  qualified  individual  to  lead  the  first  expedition  into  the 
wilderness  of  the  recently  acquired  Louisiana  Territory.  Defying 
conventional  wisdom,  he  would  eventually  choose  two  comple¬ 
mentary  leaders. 

U.S.  Army  Captains  Meriwether  Lewis  and  William  Clark  jointly 
led  the  Corps  of  Discovery  from  St.  Louis,  Missouri,  to  the  Pacific 
coast  of  the  United  States.  Both  men  held  many  key  traits  in  com¬ 
mon.  Both  possessed  strengths  the  other  lacked.  Their  partnership 
provided  leadership  that  helped  to  ensure  a  successful  and  com¬ 
prehensive  survey  of  the  land’s  majesty. 

U.S.  history  is  replete  with  enterprises  that  succeeded  due 
to  effective  partnerships.  Today,  one  of  the  most  complex  and 
demanding  relationships  may  be  found  in  the  development,  produc¬ 
tion,  fielding,  and  sustainment  of  combat  capabilities  to  the  United 
States  and  its  allies.  The  U.S.  Department  of  Defense  (DoD)  and 
the  defense  industry  are  both  engaged  in  this  pursuit,  but  does  this 
shared  interaction  form  an  effective  partnership?  Several  indicators 
seem  to  suggest  this  relationship  could  be  improved.  A  September 
2008  report  published  by  the  U.S.  Government  Accountability 
Office  (GAO)  stated: 

To  better  ensure  Warfighter  capabilities  are  delivered  when 
needed  and  as  promised,  incentives  must  encourage  a  dis¬ 
ciplined,  knowledge-based  approach,  and  a  true  partnership 
with  shared  goals  must  be  developed  among  the  depart¬ 
ment,  the  military  services,  the  Congress,  and  the  defense 
industry.  (GAO,  2008) 

Since  WWII,  the  need  for  a  true  government-defense  indus¬ 
try  partnership  has  been  clearly  established.  The  U.S.  military’s 
global  dominance  can  be  directly  attributed  to  the  technological 
superiority  gained  through  a  true  partnership  between  these  two 
complex  enterprises.  As  noted  by  the  Center  for  Defense  Informa¬ 
tion  (CDI)  (1987),  “It  is  often  forgotten  that  the  miracles  of  the 
‘Arsenal  of  Democracy’  were  the  product  of  a  government-industry 
partnership”  (CDI,  p.  36).  The  phrase  “Arsenal  of  Democracy”  was 
originated  by  President  Franklin  D.  Roosevelt  in  December  1940 
regarding  a  promise  of  assistance  to  the  British  and  the  Russians, 
then  at  war  with  Germany,  by  providing  them  with  military  supplies. 
The  phrase  also  spoke  to  the  ability  of  the  United  States  to  rapidly 
convert  its  automotive  manufacturing  capability  to  produce  military 
weapons  in  great  quantity  during  World  War  II. 

In  this  article,  we  provide  a  discussion  on  the  current  state  of 
the  shared  partnership  between  the  U.S.  DoD  materiel  acquisition 
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management  community  and  the  private  defense  industry.  We 
cite  findings  from  one  study  and  one  survey— a  study  of  program 
managers  (PMs)  serving  in  the  DoD  and  a  survey  of  PMs  work¬ 
ing  in  private  industry.  We  identify  and  discuss  what  we  believe 
to  be  five  important  facets  to  any  partnership  and  what  the  data 
suggest  about  the  relationship  between  the  DoD  and  its  industry 
partners.  Finally,  we  offer  recommendations  on  how  to  strengthen 
and  improve  existing  partnerships  and  advice  on  how  the  Defense 
Acquisition  University  (DAU)  and  the  Project  Management  Institute 
(PMI)  can  support  such  efforts. 


Current  State  of  the  Government-Defense 
Industry  Partnership 

Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics  Dr.  Ashton  Carter  recognizes  the  critical  importance  of  the 
relationship  between  the  government  and  private  industry  coun¬ 
terparts  in  materiel  acquisition  programs.  He  recently  commented: 

I  have  said  many  times. ..that  I  really  do  believe  in  the  partner¬ 
ship  between  government,  the  Department  of  Defense  and 
the  defense  industry.  The  reality  is  that  we  don't,  in  the  Gov¬ 
ernment,  build  the  weapons  systems  upon  which  our  security 
depends.  We  contract  for  them  with  the  private  sector,  and 
that  creates  a  situation  of  partnership.  (Carter,  2009) 

Dr.  Carter’s  comments  underscore  the  genuine  need  for  a  true 
government-defense  industry  partnership  to  exist  between  govern¬ 
ment  and  industry  in  the  execution  of  defense  materiel  acquisition 
programs.  However,  the  results  of  one  study  and  one  survey  con¬ 
ducted  by  the  DAU  suggest  that  DoD  and  the  defense  industry  do 
not  have  a  strong  relationship  built  on  a  true  sense  of  partnership. 
Clearly,  the  DoD  and  defense  industry  must  interface,  but  our 
research  reveals  that  something  is  clearly  lacking. 


Office  of  the  Secretary  of  Defense  Study  on  PM 
Training  and  Experience 

In  July  2009,  DAU  published  the  findings  from  a  study  commis¬ 
sioned  by  the  Director,  Portfolio  Systems  Acquisition  David  Ahern, 
on  behalf  of  the  Office  of  the  Secretary  of  Defense  (OSD).  The  study 
was  based  on  responses  from  PMs  working  in  Acquisition  Category 
I  and  II  programs.  Its  proponents  sought  to  determine  if  DoD  was 
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providing  appropriate  and  relevant  instruction  to  its  managers  and 
to  identify  any  opportunities  to  improve  the  proficiency  of  such 
individuals  (DAU,  2009,  p.  3).  The  study  included  findings  in  three 
areas:  training  topics,  training  methods,  and  acquisition  experience 
and  careers.  The  paramount  finding  listed  under  the  title  “Topics  of 
Training”  was  that  “program  managers  need  additional  training  in 
industry  practices,  including  factors  that  motivate  contractors  and 
ways  in  which  program  managers  can  use  incentives  to  achieve 
better  program  performance  for  the  government  customer”  (DAU, 
2009,  Appendix  A,  p.  13).  Given  decades  of  government-defense 
industry  partnership  in  developing,  producing,  and  fielding  war¬ 
fighter  systems,  why  is  this  a  major  area  of  concern  for  top-level 
government  PMs?  We  believe  these  findings  indicate  that  while 
government  PMs  understand  the  inherent  value  in  creating  effec¬ 
tive  partnerships  with  industry,  the  current  state  of  the  relationship 
between  these  two  enterprises  is  not  conducive  to  the  greater  levels 
of  understanding  and  partnership  that  government  PMs  desire. 


DAU  Survey  of  Defense  Industry  PMs 

In  early  2010,  DAU  conducted  a  survey  of  PMs  employed  by 
private  firms.  The  purpose  was  to  develop  an  appreciation  for  pri¬ 
vate  industry’s  perception  of  the  challenges  and  opportunities  that 
exist  in  the  shared  relationships  with  their  government  counterparts. 
The  DAU  approach  involved  a  broad  array  of  PMs  from  five  major 
defense  industry  companies.  The  results  provided  data  regarding 
the  level  of  genuine  partnership  that  currently  exists  between  DoD 
and  industry  PMs.  The  findings  from  this  survey  fell  into  five  broad 
categories:  respect,  money,  communication,  processes,  and  leader¬ 
ship.  These  five  categories  identify  what  we  believe  to  be  the  crucial 
relationship  elements  necessary  for  establishing  and  sustaining  any 
effective  and  stable  partnership,  including  the  shared  and  mutually 
beneficial  government-defense  industry  partnership. 

Respect 

Mutual  respect  is  vital  to  any  successful  partnership.  While  the 
OSD  study  of  government  PMs  yielded  no  concerns  in  this  area, 
the  DAU  survey  of  industry  PMs  pinpointed  respect  as  a  key  issue. 
Many  respondents  identified  what  they  perceived  to  be  a  commonly 
held  attitude  among  mid-career  government  employees:  Mid-career 
government  employees  do  not  see  industry  agencies  as  valued 
partners.  Instead,  these  government  employees  see  industry  as 
merely  uncommitted  vendors,  motivated  only  by  profit;  as  a  result, 
industry  must  be  managed  harshly  (Mills,  2010a).  Whether  real  or 
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perceived,  the  “we  versus  they”  mindset  exists  in  nearly  all  activi¬ 
ties  involving  the  government  and  industry,  and  is  a  major  barrier 
to  successful  partnering. 

Timing  of  the  release  of  solicitation  documents  is  another  area  of 
concern  cited  by  survey  respondents.  These  government-generated 
Requests  for  Information/Requests  for  Proposal  (RFIs/RFPs)  are 
top-priority  documents  for  private  defense  firms.  Each  RFI/RFP 
represents  a  significant  investment  of  company  time,  talent,  and 
monetary  resources  to  provide  a  timely  and  competitive  response. 
A  considerable  number  of  the  industry  PMs  surveyed  identified  the 
government’s  recurring  habit  of  releasing  RFPs  prior  to  a  major 
holiday,  along  with  a  comparatively  short  deadline  (60  days  or  less) 
for  proposal  submission.  While  this  situation  might  be  required  for 
some  programs,  the  consensus  among  the  industry  respondents  is 
that  this  is  an  all  too  common  practice.  Industry  PMs  felt  that  these 
practices  are  indicative  of  the  government-defense  industry  rela¬ 
tionship.  Moreover,  this  practice  reveals  an  inconsiderate  attitude 
toward  industry  partners. 

Money 

Differing  expectations,  attitudes,  and  purposes  for  money  are 
all  potential  sources  of  strife  between  partners.  One  senior-level 
industry  PM  responded  with  the  following  analysis  concerning  the 
importance  of  monetary  resources  for  private  industry: 

Industry  has  three  primary  concerns  when  it  comes  to 

dollars  and  cents: 

•  Acquisitions.  This  is  the  deep  fight.  Future  business 
in  the  pipeline.  This  is  where  our  business  devel¬ 
opment  process  occurs,  including  capture  and 
positioning  for  future  defense  acquisition  programs. 

•  Sales.  This  is  the  current  fight.  Plere  we  are  con¬ 
cerned  with  Return  on  Sales  and  the  amount  of 
effort  expended  over  a  specified  time  to  deliver  the 
products. 

•  Margin.  This  is  the  second  element  of  the  current 
fight.  We  address  the  question,  “How  do  we  drive 
more  profit  into  the  existing  product?  Margin  can 
be  improved  through  continuous  improvement  and 
new  technologies  to  drive  down  the  overall  product 
cost  on  the  products  being  delivered.  (Mills,  2010a) 


The  manner  and  efficiency  with  which  industry  manages  its 
money  and  achieves  acquisition,  sales,  and  margin  determine  its 
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ultimate  success.  The  findings  from  the  DAU  survey  demonstrate 
that  industry  PMs  believe  perspectives  regarding  money  are  very 
different  from  those  of  their  government  counterparts.  In  general, 
the  survey  indicates  industry  PMs  believe  government  program 
personnel  do  not  have  an  appreciation  for  the  real-world  dynamics 
with  which  private  firms  contend  in  their  effort  to  meet  acquisition, 
sales,  and  margin  demands.  Industry  PMs  expressed  some  specific 
frustrations  such  as,  “Government  does  not  understand  the  impor¬ 
tance  or  role  of  reasonable  profit  in  industry”  (Mills,  2010a),  and 
“Government  acquisition  personnel  are  generally  not  aware  of  the 
real  cost  of  goods  and  services  provided  by  industry.”  One  industry 
respondent  noted,  “Government  has  a  very  shallow  understanding 
of  industry  and  money,  overhead  rates,  wrap  rates,  fully  burdened 
costs,  etc.”  (Mills,  2010a). 

Communication 

The  ability  to  communicate  effectively  at  all  levels  of  a  partner¬ 
ship  is  crucial  for  overall  success.  Industry  respondents  provided 
different  assessments  of  government  communication  skills.  Their 
major  concerns  in  this  area  were  focused  around  two  primary  areas: 
the  poor  quality  management  of  government  solicitation  docu¬ 
ments  and  the  instability  of  customer  requirements. 

One  industry  employee  interviewed  stated,  “Government  RFPs 
are  most  often  poorly  written.  Many  are  merely  cut  and  paste 
efforts  from  earlier  RFPs,  making  them  completely  inaccurate  and 
unclear”  (Mills,  2010a).  Another  industry  employee  noted  that  the 
“government  tends  to  focus  on  improvising  the  RFP  writing  process. 
Government  expertise  in  this  area  is  very  low”  (Mills,  2010a). 

Perhaps  related  to  the  alleged  poor  quality  of  government- 
issued  RFPs  are  the  numerous  challenges  involving  requirements 
definition  and  requirements  growth— sometimes  called  require¬ 
ments  “creep.”  One  industry  respondent  cited  this  as  a  particularly 
bothersome  issue  saying,  “Requirements/scope  management  and 
managing  changes  is  the  number  one  challenge/problem  for  the 
government”  (Mills,  2010a).  Many  PMs  expressed  that  scope  and 
requirements  changes  make  a  direct,  negative  impact  on  their 
company’s  ability  to  meet  the  cost  and  schedule  terms  of  their  con¬ 
tracts  with  the  government.  One  PM  said  that  the  “government  has 
a  complete  lack  of  appreciation  for  the  impact/cost  of  changes  to 
the  program  scope,  budget,  and  schedule”  (Mills,  2010a). 

Requirements  management  challenges  have  negative  effects 
for  both  industry  and  other  government  agencies.  The  Office  of 
the  Director  of  Operational  Test  and  Evaluation  (DOT&E)  stated  in 
its  2009  annual  report: 


301 


Creating  and  Sustaining  an  Effective  Government-Defense  Industry  Partnership 


July  2011 


The  department’s  experience  indicates  that  unless  programs 
start  with  clear,  sensible,  and  rationalized  requirements, 
the  program  and  its  testing  suffer  tremendously  and  to  the 
detriment  of  our  fighting  forces.  The  DOT&E  experience  has 
been  that  no  amount  of  testing  can  compensate  or  correct 
for  unjustified  or  unrealistic  performance  expectations. 
(Gilmore,  2009,  p.  iii) 

The  two  primary  components  of  requirements  management 
are  defining  the  requirement  and  stabilizing  the  requirement.  Both 
components  must  be  effectively  managed  to  minimize  “require¬ 
ments  creep”  and  achieve  favorable  program  results.  DoD  can 
better  partner  with  industry  in  this  area  through  better  leveraging 
industry’s  Independent  Research  and  Development  (IRAD)  efforts. 
IRAD  in  industry  represents  each  company’s  efforts  to  develop  their 
technology  innovation  and  market  discrimination.  According  to 
Blakey  (2010),  this  presents  a  critical  opportunity  for  DoD  to  shape 
and  leverage  technology  development  and  the  acquisition  process. 

DoD  must  identify  future  technology  requirements  so  that 
industry  can  plan  its  IRAD  investments.  Requirements  for 
new  systems  must  be  based  on  well-understood  technolo¬ 
gies  and  stable  product  rates,  to  allow  industry  to  develop 
and  build  systems  efficiently  with  the  right  contract  type.  For 
example,  one  company  recently  consolidated  its  shipbuild¬ 
ing  operations  and  is  considering  selling  the  unit  outright  in 
response  to  its  vision  of  the  future  business  environment, 
(p.  68) 

This  approach,  when  implemented  effectively,  provides  industry 
with  much  needed  and  desired  stability.  Clearly  defined  and  stable 
requirements  remain  critical  to  program  success. 

Processes 

Another  important  facet  in  establishing  effective  partnerships 
is  an  understanding  of  and  deference  to  the  key  processes  of  one’s 
partner.  As  cited  earlier,  government  PMs  acknowledge  the  benefit 
of  better  understanding  industry  practices  and  processes.  Like¬ 
wise,  industry  PMs  interviewed  identified  a  need  for  an  improved 
understanding  of  government  materiel  acquisition  management 
processes.  “Industry,”  one  respondent  said,  “needs  to  better  under¬ 
stand  the  government  [DoD  Instruction]  5000.02  processes  in 
order  to  work  more  effectively  with  the  government”  (Mills,  2010a). 
Another  industry  PM  provided  this  insight: 
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Some  companies  have  strong  PM  culture  and  PM  training 
programs.  Industry  standard  credentials  [PMI  Project  Man¬ 
agement  Professional,  for  example]  bring  PM  skills  to  the 
table,  but  industry  needs  to  understand  the  DoDI  5000.02 
process  in  order  to  work  more  effectively  with  government. 
(Mills,  2010a,  p.6) 

Industry  PMs  strongly  believe  that  mutual  training  opportunities 
represent  the  best  avenue  to  better  understand  and  improve  the 
shared  processes  of  government  and  industry.  Specifically,  some 
respondents  noted  that  DAU  courses  have  the  ability  to  overcome 
the  shortcomings  of  both  government  and  industry  in  understand¬ 
ing  the  acquisition  process. 

What  prevents  reciprocal  training  among  employees  of  the 
government  and  the  defense  industry?  One  challenge  is  that 
industry  often  does  not  incentivize  personnel  to  attend  classes 
because  career  progression  is  not  tied  to  the  training  and  experi¬ 
ence  afforded  by  government  courses.  Classes  of  this  type  are 
usually  deducted  from  the  company’s  “overhead”  funds,  which  are 
generally  very  limited.  Finally,  government  acquisition  personnel 
have  priority  for  admittance  in  government  courses,  thus  limiting 
the  ability  of  private  employees  to  participate.  This  is  especially 
important  in  light  of  the  increased  Defense  Acquisition  Workforce 
employee  population  projected  for  the  near-  to  mid-term. 

Leadership 

Leadership  is  the  most  crucial  component  needed  for  establish¬ 
ing  effective  partnerships.  Leadership  can  be  described  as  the  art 
of  influencing  people.  Leadership  is  necessary  for  the  application 
and  management  of  all  essential  facets  for  the  creation  of  genuine 
partnerships. 

Respondents  to  both  the  OSD  study  and  the  DAU  survey  iden¬ 
tified  leadership  as  a  key  focus  area.  According  to  the  OSD  study, 
in  the  areas  of  acquisition,  experience,  and  careers,  “PMs  need 
mentors  and  senior  advisory  teams  to  assist  them  in  dealing  with 
particularly  complex  challenges  on  major  acquisition  problems” 
(DAU,  2009,  p.  24).  In  this  area,  one  government  PM  stated  spe¬ 
cifically,  “The  best  preparation  for  a  future  program  manager  is 
working  with  extraordinary  leaders-mentors”  (DAU,  2009,  p.  24). 

In  the  DAU  survey  of  industry  PMs,  respondents  stated  that 
the  government  leadership  at  the  executive  levels  was  sound,  but 
they  expressed  concern  about  junior  employees.  “Although  the 
PM  leadership  and  competency  tends  to  be  good,  this  is  not  so  at 
the  lower  levels”  (Mills,  2010a).  Interestingly,  industry  PMs  perceive 
a  cultural  shift  occurring  within  the  government  acquisition  com- 
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munity.  One  PM  observed  “a  younger,  more  aggressive  crowd  that 
lacks  understanding  and  experience  [is  emerging]  in  acquisition.  For 
these  younger  personnel,  failure  is  frightening”  (Mills,  2010a).  These 
new  members  of  the  Defense  Acquisition  Workforce  are  generally 
very  talented,  but  recognize  that  their  lack  of  experience  is  a  chal¬ 
lenge  that  must  be  overcome.  The  OSD  study  and  the  DAU  industry 
survey  reveal  that  deliberate,  proactive  engagement  between  more 
seasoned  government  acquisition  professionals  and  their  younger, 
less  experienced  colleagues  would  improve  overall  partnerships 
shared  by  the  government  and  industry. 


Solutions  and  Recommendations 

An  effective  DoD-defense  industry  partnership  is  not  only 
attainable,  but  will  improve  overall  acquisition  outcomes.  The  Figure 
represents  a  proposed  model  for  the  creation  of  effective  partner¬ 
ships  between  government  program  offices  and  private  defense 
firms.  Collective  experience  and  observation  demonstrate  that  the 
primary  desire  and  expectation  of  industry  firms  from  government 
program  offices  is  flexibility.  In  other  words,  government  PMs  look 
for  private  industry  to  provide  required  capability  as  close  to  the 
original  schedule  and  cost  estimates  as  possible,  regardless  of 
unforeseen  events  and  changes.  Private  firms  seek  stability  from 
their  government  partners.  In  other  words,  private  industry  seeks  a 
measure  of  confidence  with  regard  to  the  level  of  current  and  future 
work  they  will  be  employed  to  perform  for  the  government.  Stabil¬ 
ity  enables  industry  to  manage  subcontract  relationships,  adjust 
personnel  staffing  levels,  and  forecast  company  performance  for 
the  benefit  of  their  investors. 


FIGURE.  EFFECTIVE  PARTNERSHIP  BETWEEN  GOVERNMENT 
AND  INDUSTRY 
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Complete  flexibility  for  the  government  and  complete  stability 
for  industry  are  not  attainable.  Each  entity  must  help  achieve  the 
other’s  requirements  for  the  partnership  to  be  effective.  We  sug¬ 
gest  that  the  previously  discussed  five  components  for  an  effective 
partnership  serve  as  the  center  of  mass  through  which  government 
and  industry  should  seek  to  interface. 


DAU  Support 

Within  DAU,  government  and  industry  PMs  have  at  their  disposal 
an  extensive  program  management  and  acquisition  management 
tool.  While  the  primary  focus  of  DAU  training  assets  is  to  enable 
success  in  managing  DoD  acquisition  programs,  we  believe  the 
university  could  also  provide  direct  support  in  the  effort  to  establish 
and  sustain  effective  partnerships. 

High-quality  acquisition  training  is  available  in  all  of  the  func¬ 
tional  areas  that  support  DoD  materiel  acquisition  programs.  DAU’s 
resources  extend  well  beyond  the  classroom  and  include  a  signifi¬ 
cant  online  presence.  A  quick  review  of  the  DAU  website  (http:// 
www.dau.mil)  and  its  related  learning  resources  reveals  a  large  num¬ 
ber  of  educational  opportunities  targeted  to  both  government  and 
industry  employees.  The  Defense  Acquisition  Guidebook  (DAG)  is  a 
great  example  of  a  valuable  resource  available  to  industry  and  gov¬ 
ernment  acquisition  personnel  at  any  time.  The  DAG  (https://dag. 
dau.mil/Pages/Default.aspx)  provides  a  graphical  and  user-friendly 
portal  of  DoD  acquisition  best  business  practices,  acquisition  policy, 
and  lessons  learned.  Another  excellent  source  of  acquisition  training 
for  both  DoD  and  industry  personnel  is  DAUs  Continuous  Learning 
website  (http://www.dau.mil/clc/default.aspx),  which  hosts  over  175 
Continuous  Learning  Modules  covering  critical  topics  that  support 
all  of  the  12  Acquisition  Workforce  functional  areas.  These  learning 
assets  are  also  available  to  both  DoD  and  industry  personnel  24 
hours  a  day. 


Project  Management  Institute  Support 

The  Project  Management  Institute  (PMI)  also  offers  training  and 
certification  opportunities  to  government  and  industry  personnel. 
Like  DAU-sponsored  training,  we  believe  this  training  would  be  very 
effective  in  the  creation  and  management  of  partnerships  between 
the  two  enterprises.  This  is  particularly  true  since  PMI’s  suite  of 
globally  recognized  processes  will  significantly  affect  the  worldwide 
defense  industry  in  the  future. 
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The  field  of  program  management  in  private  industry  is  guided 
by  both  doctrine  and  best  practices.  PMI’s  Project  Management 
Body  of  Knowledge  ( PMBOK ®),  as  embodied  in  the  PMBOK ®  Guide, 
serves  as  the  repository  for  both  industry-developed  doctrine  and 
best  practices  in  program  management  (PMI,  2008).  The  PMBOK  is 
the  industry  standard  for  program  management  doctrine  and  best 
practices.  It  represents  the  PM  approach  embraced  by  our  industry 
partners.  PMI  plays  a  key  role  by  serving  as  the  granting  authority 
for  several  American  National  Standards  Institute-based  credentials. 
These  include: 

•  Certified  Associate  in  Project  Management  (CAPM®)— 
for  integrated  product  team  leaders  and  members 

•  Project  Management  Professional  (PMP®)— for  project/ 
program  managers 

•  Program  Management  Professional  (PgMP®)— for  pro¬ 
gram/portfolio  managers 

•  Risk  Management  Professional  (PMI-RMP®)— for  risk 
managers 

•  Scheduling  Professional  (PMI-SP®)— for  scheduling 
managers 

Significant  commonality  exists  between  the  PMBOK®  and  gov¬ 
ernment  acquisition  management  doctrine.  Like  Lewis  and  Clark, 
each  body  of  knowledge  complements  the  other.  This  commonality 
provides  an  opportunity  for  government  acquisition  professionals  to 
bridge  the  knowledge  gap  through  the  study  and  accreditation  of 
PMI-sponsored,  PMBOK®-based  credentials.  Government  PMs  who 
obtain  PMI  credentials  gain  a  better  understanding/perspective  of 
industry  program  management  processes  and  best  practices. 

The  process  of  promoting  industry  standard  credentials  as 
a  career  progression  option  for  Defense  Acquisition  Workforce 
employees  would  accomplish  multiple  objectives.  First,  it  would  help 
create  a  better  channel  of  communication  between  government 
and  industry  personnel  by  creating  a  common  understanding.  By 
focusing  on  and  understanding  the  industry  standard  for  project 
management,  a  common  context  for  project  management  discus¬ 
sions  would  exist.  The  use  of  earned  value  management  provides 
a  good  example  of  a  common  process  or  understanding.  Second, 
PMI’s  Aerospace  and  Defense  Specific  Interest  Group  (A&D  SIG) 
could  supplement,  to  some  degree,  DAU’s  workforce  training  at 
large,  given  the  capacity  limitations  discussed  earlier.  Lastly,  pro¬ 
moting  the  value  of  industry  standards  for  project  management 
would  demonstrate  a  commitment  from  an  organization’s  leader¬ 
ship  to  the  professional  development  of  the  individual.  The  most 
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important  resource  required  for  attaining  success  in  government 
acquisition  programs  is  a  well-trained  and  well-led  workforce,  com¬ 
posed  of  both  government  and  industry  employees. 


Recommended  Areas  for  Consideration 

Along  with  the  contributions  to  effective  partnership  available 
through  DAU  and  PMI  training,  we  recommend  consideration  of  the 
following  measures: 

Recommendation  No.  1— Training  With  Industry  (TWI) 

Currently,  limited  TWI  opportunities  are  afforded  DoD  acquisi¬ 
tion  personnel.  The  OSD  study  found  that  “adoptfing]  the  training 
with  industry  program  more  widely.. .[would]  be  important  in 
improving  management  of  acquisition  programs”  (DAU,  2009, 
Comment  E61C).  We  agree  with  this  assessment  and  recommend 
that  OSD  and  the  defense  industry  PM  offices  collaborate  to 
increase  the  number  of  TWI  offerings  made  available  each  year  to 
acquisition  professionals. 

Recommendation  No.  2— Incentivize  DAU  Course  Attendance 

In  an  effort  to  increase  opportunities  and  funding  for  contrac¬ 
tor  attendance  at  DAU  courses,  we  recommend  that  government 
program  offices  request  industry  partners,  in  their  contract  propos¬ 
als  to  the  government,  provide  the  names  and  cost  estimates  for  a 
finite  number  of  their  personnel  to  complete  DAU  training.  Specifi¬ 
cally,  these  would  be  industry  personnel  working  in  support  of  the 
government  contract.  This  arrangement  would  allow  the  contractor 
firm  to  directly  charge  for  the  training  of  their  personnel,  thus  elimi¬ 
nating  the  concern  of  overextending  vital  overhead  funds.  In  return, 
the  government  project  office  would  be  supported  by  personnel 
equipped  with  better  knowledge  of  the  DoD  materiel  acquisition 
processes.  Government  project  offices  could  secure  the  necessary 
funding  via  the  annual  DoD  planning,  programming,  budgeting,  and 
execution  process. 

Recommendation  No.  3— Update  DAU  Course  Content 

Providing  DAU  students  with  a  better  appreciation  for  the  reali¬ 
ties  of  private  defense  industry  funding  challenges  would  greatly 
assist  with  efforts  to  facilitate  effective  partnerships  among  govern¬ 
ment  and  industry.  The  DAU-South  Region  has  already  developed 
a  new  section  for  its  Intermediate  Systems  Acquisition  (ACQ  201B) 
course  curriculum,  in  which  students  are  presented  the  basic  instruc¬ 
tion  on  direct,  indirect,  and  loaded  rate  personnel  charges.  This  is  a 
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line  of  teaching  that  can  and  should  be  incorporated  into  other  DAU 
courses  in  all  functional  areas. 

Recommendation  No.  4— Develop  New  Industry-Specific 
Course  Content 

DAU  recently  developed  a  new  course  titled,  “Understanding 
Industry.”  This  course  was  successfully  piloted  with  the  Senior  Ser¬ 
vice  College  Fellowship  students  at  the  DAU  South  Region  campus 
in  September  2010.  The  focus  of  this  new  DAU  offering  is  to  provide 
the  Defense  Acquisition  Workforce  significant  insight  into  how  our 
industry  partners  function  and  support  DoD  materiel  acquisition 
programs.  This  course  is  a  comprehensive  2.5  day  offering  that 
educates  students  on  the  key  aspects  of  our  industry  partner’s  busi¬ 
ness  processes  and  challenges  them  with  a  Capstone  exercise  as 
well.  Overall  results  of  the  initial  pilot  offering  were  very  favorable. 
Additional  course  refinement  continues. 

Recommendation  No.  5— Increased  DAU  Engagement  of  Industry 

In  addition,  DAU  could  provide  regular  engagement  opportuni¬ 
ties  with  the  employees  and  leadership  of  private  defense  industry 
organizations  to  help  them  better  understand  the  DoDI  5000.02 
and  associated  processes  (DoD,  2008).  DAU  already  provides  annual 
seminars  to  the  National  Defense  Industry  Association  and  its  mem¬ 
ber  organizations.  Regular  affiliations  of  this  sort  between  DAU  and 
industry  would  be  of  significant  value  to  the  overall  effort  to  establish 
and  maintain  effective  government-defense  industry  partnerships. 


Conclusions 

Like  Lewis  and  Clark,  the  government  and  defense  industry  need 
to  foster  a  true  and  sustained  partnership.  Other  leaders  in  the  DoD 
acquisition  management  community  agree.  Marine  Colonel  Michael 
Micucci,  project  manager  for  Light  Armored  Vehicles,  noted  in  the 
Marine  Corps  Systems  Command  News  (Johnson-Miles,  2009): 

Cost,  schedule,  and  performance  requirements  are  definitely 
important,  and  meeting  them  is  key  to  program  success; 
however,  they  really  represent  the  lowest  common  denomi¬ 
nator  in  the  professional  partnership  formed  by  the  defense 
acquisition  professionals  and  industry.. .with  this  in  mind,  we 
should  explore  establishing  expectations  for  industry  as  a 
full  partner  in  every  success,  (p.  1) 
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Dr.  Carter  also  emphasized  the  importance  of  the  government- 
defense  industry  relationship.  “I  am  not  a  believer  that  the  defense 
industry  is  the  enemy;  they  are  our  partners.  We  can’t  arm  and 
defend  the  country  without  private  industry”  (Mills,  2010b).  Con¬ 
certed  efforts  by  both  government  and  industry  to  engage  the  five 
facets  identified  in  this  discourse  will  improve  both  the  flexibility 
industry  can  provide  to  the  government  and  the  stability  govern¬ 
ment  can  provide  to  industry.  This  “win-win”  arrangement  will 
certainly  be  beneficial  for  government  operations  and  for  industry 
bottom  lines.  Even  more  important  than  these  benefits,  effective 
partnerships  between  government  and  industry  will  provide  U.S. 
and  allied  warfighters  with  better  capabilities  delivered  in  a  more 
timely  and  cost-effective  manner. 
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The  Defense  Acquisition  Workforce  is  getting  younger,  and  its 
educational  expectations  include  using  advanced  and  innova¬ 
tive  learning  technologies.  The  Defense  Acquisition  University 
(DAU)  has  fully  embraced  this  generational  trend  and  has 
partnered  with  several  institutions  to  conduct  research  on 
Advanced  Learning  Technologies,  or  ALT.  One  such  partnership 
is  with  George  Mason  University’s  Instructional  Technology 
Immersion  Program.  The  partnership’s  goal  was  to  examine 
DAU’s  current  learning  assets  and  identify  processes  and 
methods  for  utilizing  innovative  learning  technology  designs. 

This  article  summarizes  this  effort  and  describes  the  resulting 
online  performance  support  tool  called  LATIST  (Learning  Asset 
Technology  Integration  Support  Tool)  developed  to  facilitate 
the  understanding,  selection,  and  integration  of  ALT  by  DAU 
faculty  and  staff. 
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As  of  September  30,  2008,  the  Defense  Acquisition  Workforce 
was  just  under  126,000  personnel;  an  estimated  76  percent  of  that 
2008  workforce  was  classified  as  baby  boomers— the  majority  of 
which  are  now  approaching  retirement  (DoD,  2007).  Waiting  in  the 
wings  is  the  gamer  generation.  As  these  gamers  enter  the  workforce 
and  subsequently  become  students  at  the  Defense  Acquisition 
University  (DAU),  they  do  so  with  expectations  about  their  educa¬ 
tional  environment.  The  new  generation  finds  classroom  settings 
that  have  little  visual  stimulation,  passive  lectures,  and  ineffective 
or  unengaging  use  of  integrated  technology  (Kapp,  2007).  DAU 
recognizes  this  change  in  expectations  and  wants  to  enhance  its 
learning  assets  and  prepare  for  the  future  by  smartly  integrating 
available  technologies  (Anderson,  Hardy,  &  Leeson,  2008).  For 
example,  DAU  is  already  using  innovative  approaches  to  develop 
games  and  simulations  designed  to  improve  performance  outcomes 
(Sanchez,  2009).  However,  the  pace  from  invention  to  production 
to  maturity  of  these  new  technologies  is  shortening  dramatically 
(Oehlert,  2009). 

Keeping  pace  with  these  new  technologies,  their  capabili¬ 
ties,  and  impact  as  possible  tools  for  training  and  education  is  an 
ongoing  challenge.  As  such,  DAU  partnered  with  the  Instructional 
Technology  Immersion  Program  at  George  Mason  University  (GMU) 
to  conduct  comprehensive  research  on  Advanced  Learning  Tech¬ 
nologies  (ALT)  in  order  to  determine  how  best  to  integrate  such 
technologies  into  its  learning  assets.  Specifically,  the  purpose  of 
this  research  was  to: 

Examine  current  DAU  training  programs  and  learning 
products  and  identify  processes  and  methods  for  utiliz¬ 
ing  innovative  learning  technology  designs  and  delivery 
tools  within  the  current  DAU  learning  modalities  in  order  to 
improve  the  effectiveness  and  efficiency  of  current  offerings. 

This  article  summarizes  this  effort  including  a  description  of  the 
instructional  design  model  used  to  guide  this  research;  the  three- 
step  performance  analysis  process  conducted  to  explicitly  define 
DAU  technology  integration  needs;  and  the  performance  support 
tool,  LATIST  (Learning  Asset  Technology  Integration  and  Support 
Tool),  developed  to  facilitate  the  understanding,  selection,  and 
integration  of  ALT  by  DAU  faculty  and  staff. 
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FIGURE  1.  INTEGRATIVE  LEARNING  DESIGN  FRAMEWORK 
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The  overarching  methodology  used  to  guide  this  research  effort 
relied  on  a  research-driven  approach  known  as  the  Integrative 
Learning  Design  Framework  (ILDF)  for  E-Learning  (Dabbagh  & 
Bannan-Ritland,  2005).  The  ILDF  for  e-Learning  is  an  instructional 
design  model  that  provides  a  systematic  framework  for  the  develop¬ 
ment  of  learning  assets  and  products  based  on  sound  pedagogy, 
iterative  evaluation,  and  the  socio-cultural  context  of  the  learning 
organization  (Figure  1). 

The  ILDF  is  a  comprehensive  and  flexible  model  that  draws 
heavily  from  the  iterative  nature  of  traditional  systematic  processes 
of  instructional  design  and  can  be  applied  in  multiple  settings.  Spe¬ 
cifically,  the  ILDF  consists  of  three  phases: 

•  Exploration— Investigating  and  documenting  relevant 
information  related  to  the  instructional  or  training  set¬ 
ting,  including  stakeholders’  individual  and  collective 
beliefs  on  learning  and  solicited  information  from  all 
involved  in  the  instructional  or  training  situation. 

•  Enactment— Mapping  information  gathered  in  the  explo¬ 
ration  phase  about  learning  processes,  content,  and 
context  to  existing  pedagogical  models,  considering 
the  characteristics  of  the  selected  model(s)  to  identify 
and  implement  effective  instructional  strategies  using 
technology. 

•  Evaluation— Determining  the  purpose,  desired  results, 
and  methods  of  evaluation  of  an  online-  or  technology- 
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supported  learning  design,  incorporating  formative 
evaluation  and  revision  cycles  that  result  in  effective 
implementation  and  informative  results. 

In  this  research  effort,  the  first  phase  of  the  ILDF— Exploration- 
consisted  of  conducting  a  Performance  Analysis  (PA).  Through 
formal  and  informal  data-gathering  techniques  and  in  collaboration 
with  DAU  stakeholders,  the  GMU  Immersion  team  implemented  a 
three-step  PA  process  to  (a)  identify  relevant  roles,  responsibilities, 
and  processes,  (b)  define  factors  for  successful  performance,  and 
(c)  propose  possible  solutions  for  effective  instructional  design.  The 
three-step  PA  process  for  this  project  is  explained  in  the  next  section. 


Performance  Analysis  Process 

The  three-step  PA  process  began  with  a  front-end  analysis, 
followed  by  an  extant  data  analysis,  and  concluded  with  a  needs 
assessment. 

Front-End  Analysis 

The  front-end  analysis  provided  a  preliminary  understanding 
of  DAU’s  education  and  training  program.  The  analysis  relied  on 
informal  data-gathering  techniques  by  examining  relevant  docu¬ 
ments  such  as:  the  DAU  Course  Catalog ;  the  Defense  Acquisition 
Review  Journal  (now  the  Defense  Acquisition  Research  Journal ); 
the  Defense  AT&L  magazine;  the  AT&L  Human  Capital  Strategic 
Plan ;  and  DAU  Directive  709,  which  addresses  the  Learning  Asset 
Management  Program.  Additionally,  key  stakeholder  meetings 
explored  DAU’s  learning  asset  development  processes,  roles,  and 
responsibilities.  The  analysis  indicated  DAU’s  educational  philoso¬ 
phy  is  exemplified  through  its  Performance  Learning  Model  (PLM) 
(DAU,  2010,  p.  28). 

The  three  pillars  of  the  PLM  (Career  Development,  Job  Perfor¬ 
mance,  Executive  &  Leadership)  represent  the  three  main  types  of 
training  and  development  that  the  Defense  Acquisition  Workforce 
receives  (DAU,  2010,  p.  28).  Training  courses,  continuous  learning 
modules,  performance  support,  and  knowledge  sharing  capabilities 
are  the  primary  methods  utilized  to  build  DAU’s  learning  environ¬ 
ment  (DAU,  2008a).  These  are  all  considered  to  be  learning  assets. 
Formal  courses  can  be  delivered  on-site,  online,  or  in  a  hybrid 
approach.  Informal  support  is  provided  through  a  variety  of  tools. 
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For  example,  DAU  provides  an  extensive  Community  of  Practice 
(CoP)  to  help  “extend  the  reach  of  Subject  Matter  Experts  (SMEs) 
supporting  the  workforce”  (Garcia  &  Dorohovich,  2005,  p.  21).  DAU 
also  supports  a  virtual  library,  an  electronic  Defense  Acquisition 
Guidebook ,  an  Ask  a  Professor  service  as  well  as  its  own  version  of 
Wikipedia  called  Acquipedia. 

The  front-end  analysis  also  revealed  that  similar  to  other  learning 
organizations,  DAU  faces  several  challenges  and  constraints  and  has 
multiple  motivational  drivers  when  it  comes  to  integrating  technol¬ 
ogy  into  its  learning  assets.  Challenges  included  promoting  new 
technologies  that  may  exceed  available  bandwidth  or  compromise 
security  standards  for  some  units;  constraints  included  working 
within  an  existing  learning  management  system  and  well-defined 
certification  classes;  and  drivers  included  supportive  leadership, 
access  to  cutting-edge  technology,  and  commitment  to  advanced 
technology  research.  Armed  with  this  preliminary  knowledge,  an 
extant  data  analysis  was  performed  to  continue  the  PA  process. 

Extant  Data  Analysis 

The  extant  data  analysis  revealed  that  as  the  provider  of  acquisi¬ 
tion  training  for  the  DoD  community,  DAU  supports  many  external 
stakeholders  and  customers  including  the  Services  and  other 
Defense  Agencies.  Within  DAU,  the  Global  Learning  Technology 
Center  (GLTC)  is  leading  the  research  on  ALT  for  DAU.  The  DAU 
Learning  Capabilities  and  Integration  Center  (LCIC)  is  responsible 
for  the  development  and  management  of  training  requirements 
and  content,  which  traverses  11  functional  areas.  To  support  these 
multiple  roles  and  responsibilities,  DAU  uses  an  Integrated  Product 
Team  (IPT)  approach  to  develop  its  learning  assets  (DAU,  2008b). 
Commonly,  a  learning  asset  IPT  is  composed  of  SMEs,  a  gaming/ 
simulation/technology  representative,  an  instructional  systems 
designer,  and  a  Performance  Learning  Director  (PLD).  Depending 
on  the  type  of  learning  asset,  the  use  of  these  roles  may  vary.  In 
the  case  of  a  training  course,  the  SME  develops  the  content  docu¬ 
ment;  the  instructional  systems  designer  provides  recommendations 
related  to  learning  outcomes  and  helps  define  terminal  and  enabling 
objectives;  the  gaming/simulation/technology  representative  makes 
recommendations  on  the  use  of  such  technologies  in  the  learning 
asset;  while  the  PLD  oversees  and  participates,  as  appropriate. 
Additionally,  functional  area  representatives  have  the  opportunity 
to  participate  and  evaluate  pilot  courses.  Once  a  course  is  launched, 
a  manager  is  assigned  to  oversee  postproduction  activities  such  as 
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addressing  postcourse  surveys  that  capture  student  opinions  as  well 
as  addressing  the  ever-changing,  sporadic  requirements  that  affect 
content  currency,  relevancy,  and  accuracy. 

At  any  point  in  the  learning  asset  development  life  cycle  process, 
the  recommendation  to  use  learning  technologies  can  be  made. 
However,  several  factors  need  to  be  taken  into  consideration  such 
as  recognizing  that  duplicate  learning  assets  exist  deliberately  to 
accommodate  bandwidth  constraints.  Another  factor  is  speed-to- 
market,  which  involves  balancing  the  subject  matter  vetting  process 
against  timeliness  of  content  release.  Such  factors  greatly  impact 
implementing  technology  recommendations. 

The  extant  data  analysis  also  revealed  that  DAU  needed  a  formal 
vehicle  or  process  to  effectively  diffuse  GLTC  research  findings  on 
how  to  select  and  integrate  pedagogically  appropriate  ALT  into 
the  life  cycle  of  a  learning  asset.  Furthermore,  the  analysis  revealed 
that  DAU  faculty  and  staff  could  benefit  from  an  area  or  “sand¬ 
box”  to  practice  and  become  familiar  with  ALT.  Based  on  these 
findings,  a  potential  strategy— namely  a  conceptual  framework 
comprised  of  four  components— was  developed  to  address  these 
needs  (Figure  2).  Next,  a  needs  assessment— the  third  step  in  the 


FIGURE  2.  CONCEPTUAL  FRAMEWORK 
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PA  process— was  conducted  to  validate  these  findings  and  refine 
the  framework. 

Needs  Assessment 

The  needs  assessment  required  more  formal  data-gathering 
techniques.  As  such,  structured  interviews  and  an  online  survey  were 
conducted  to  further  the  current  understanding  of  roles,  decisions, 
and  processes  associated  with  the  integration  of  ALT  into  DAU’s 
learning  assets.  The  structured  interviews  consisted  of  nine  sequen¬ 
tial  questions  posed  to  each  of  12  DAU  participants,  including  LCIC 
leadership,  a  knowledge  project  officer,  a  center  director,  and  eight 
representatives  from  the  DAU  Capital  and  Northeast  Region.  The 
goal  of  the  interviews  was  to  provide  qualitative  data  about  the  dif¬ 
ferent  roles  that  DAU  personnel  assume  in  the  process  of  selecting 
and  integrating  ALT  into  DAU’s  learning  assets  as  well  as  identify¬ 
ing  factors  that  influence  ALT  selection  and  adoption.  At  the  end  of 
each  interview,  the  conceptual  framework  (Figure  2)  was  presented 
to  gather  feedback  on  its  usefulness  as  a  vehicle  or  tool  to  facilitate 
technology  integration.  The  online  survey  was  informed  by  the 
results  of  the  structured  interviews  and  accessed  a  wider  audience 
to  collect  a  larger  data  set.  The  survey  consisted  of  six  forced  choice 
answers  targeting  demographic  data,  four  Likert-scale  questions 
that  targeted  respondents’  familiarity  with  ALT,  and  one  Likert-scale 
question  addressing  the  likelihood  that  a  respondent  would  use  an 
online  framework,  namely  the  conceptual  framework. 

The  interview  data  resulted  in  591  comments,  which  were 
aggregated  across  questions  and  analyzed  qualitatively.  Descrip¬ 
tive  analysis  was  conducted  on  the  survey,  which  consisted  of 
34  responses.  The  interview  data  confirmed  that  LCIC  and  GLTC 
personnel  consider  and  provide  recommendations  regarding  ALT 
integration  at  various  points  in  the  learning  asset  development 
process,  and  that  there  is  significant  interest  in  incorporating  ALT 
in  a  pedagogically  appropriate  manner  that  would  lead  to  improved 
learning  outcomes.  Additionally,  20  factors  influencing  the  selec¬ 
tion  and  integration  of  ALT  were  culled  from  the  interviews.  These 
factors  were  provided  in  the  online  survey  for  rank  ordering.  The 
results  of  the  survey  revealed  that  the  top  five  factors  were  long¬ 
term  revision  feasibility,  content  stability,  bandwidth,  ability  to 
replicate  across  regions,  and  development  cost.  Survey  respondents 
also  indicated  that  decisions  regarding  ALT  integration  are  made 
across  all  roles  and  job  titles  within  GLTC  and  LCIC,  reflective  of  the 
IPT  approach. 

With  respect  to  the  conceptual  framework,  half  the  interview¬ 
ees  expressed  interest  in  it  as  both  a  decision  tool  that  guides  ALT 
selection,  and  training  support  on  how  and  why  to  use  different  tech- 
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nologies;  83  percent  of  the  survey  respondents  indicated  they  would 
use  it.  These  findings  further  validated  interviewee  suggestions  that 
developing  a  systematic  process  for  technology  integration  would 
improve  the  effectiveness  and  efficiency  of  learning  asset  develop¬ 
ment.  With  the  completion  of  the  needs  assessment,  and  hence  the 
PA,  the  GMU  Immersion  team  proceeded  to  the  enactment  and  eval¬ 
uation  phases  of  the  ILDF  to  formalize  the  design  and  development 
of  the  proposed  conceptual  frameworks  as  a  vehicle  for  disseminat¬ 
ing  the  latest  research  on  ALT  and  facilitating  the  integration  of  ALT 
into  DAU’s  learning  asset  development  processes. 


Prototype  Development 

In  collaboration  with  DAU  stakeholders,  the  conceptual  frame¬ 
work  was  named  the  Learning  Asset  Technology  Integration 
Support  Tool  (LATIST).  LATIST  would  be  developed  as  an  area 
where  users  could  explore  the  latest  research  on  ALT;  as  a  decision 
tool  that  would  guide  the  selection  of  ALT  based  on  users’  instruc¬ 
tional  needs  and  contextual  factors;  and  as  an  application  area 
where  users  could  practice  implementing  ALT.  Although  the  initial 
vision  for  the  conceptual  framework  included  a  user  community 
for  information  sharing  and  collaboration,  this  was  not  pursued  for 
development  since  DAU  has  an  extensive  CoP  infrastructure.  The 
specifics  of  the  content  and  functionality  of  the  three  components 
of  LATIST  would  be  determined  through  a  usage-centered  design 
process  described  in  the  next  section. 

Usage-Centered  Design  Process 

A  usage-centered  design  process  is  commonly  used  in  software 
development  and  focuses  on  identifying  user  needs  to  develop  a 
product  that  allows  users  to  fulfill  their  needs  in  an  easy,  effective, 
and  efficient  way  (Constantine  &  Lockwood,  1999).  Through  content 
and  task  analysis,  user  needs  were  confirmed,  tasks  necessary  for 
users  to  obtain  optimal  benefit  while  using  LATIST  were  identified, 
and  content  requirements  were  established  based  on  projected  user 
scenarios  and  use  cases.  As  a  result,  it  was  determined  that  LATIST 
would  consist  of  three  components  that  would  allow  the  user  to: 

1.  Explore  what  the  research  says  about  a  technology 
such  as  advantages  and  disadvantages; 

2.  Select  a  best  technology  for  user  conditions  such  as 
learning  objectives  and  bandwidth  constraint;  and 

3.  Review  and  learn  how  to  apply  a  selected  technology. 
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Additionally,  the  results  of  the  content  and  task  analysis  revealed 
that  LATIST  would  best  be  utilized  as  an  electronic  performance 
support  system  (EPSS).  An  EPSS  is  an  easily  accessible,  integrated 
electronic  environment  that  provides  immediate,  individualized 
support  so  employees  can  perform  their  duties  with  minimal  inter¬ 
vention  by  others  (Dickelman,  2004).  A  performance  support  SME 
recommended  that  for  maximum  effectiveness,  information  in 
LATIST  should  be  explicit,  accessible,  and  usable;  and  confirmed 
that  the  three  components  of  LATIST  represented  a  successful 
integration  of  process  and  knowledge. 

Design  Requirements 

LATIST  was  to  support  information  access  in  an  anytime,  any¬ 
place  environment,  including  mobile.  Therefore,  LATIST  was  to  be 
browser-based,  quickly  accessible  from  the  Internet.  The  design 
requirements  were  derived  specifically  from  use  cases  that  delin¬ 
eated  the  features  and  functionality  of  the  EPSS  based  on  user 
perspectives.  These  included  the  capability  for  users  to  print,  save, 
search,  and  share  content;  create  a  personal  space  of  notes  and 
personally  rated  content;  upload  content;  navigate  across  compo¬ 
nents  based  on  a  selected  technology;  and  access  support  features 
such  as  Help  and  Dictionary.  Given  the  dynamic  nature  of  the  “select 
best  technology”  component  and  the  need  for  searchable  research 
resources,  a  back-end  database  was  necessary  to  manage  the 
content.  Hence,  a  Content  Management  System  (CMS)  was  used 
to  build  LATIST.  Based  on  these  requirements  and  discussions  with 
DAU  stakeholders  and  information  technology  specialists,  the  CMS 
WordPress  was  selected  to  build  the  core  framework  for  LATIST. 
WordPress  is  an  open-source  blog  publishing  CMS  application  pow¬ 
ered  by  an  open-source  server-side  scripting  language  known  as 
Hypertext  Preprocessor  (PHP)  and  by  MySQL— a  relational  database 
management  system  that  can  also  be  used  for  content  manage¬ 
ment.  LATIST  navigation  was  designed  to  be  intuitive  and  to  address 
these  four  focal  use  cases: 

•  A  user  who  is  not  familiar  with  what  technology  can  do 
in  a  teaching  and  learning  context  and  wants  to  review 
what  the  research  says  about  technology; 

•  A  user  who  has  a  known  learning  outcome  (instructional 
objective)  and/or  contextual  factors  and  wants  to  see 
what  technologies  might  be  beneficial  for  that  instance; 

•  A  user  who  wants  to  learn  how  to  apply  a  technology 
and  practice  those  steps;  and 
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•  A  user  who  has  been  directed  to  use  a  particular  tech¬ 
nology  and  wants  to  learn  what  the  research  says  about 
the  technology  and  how  to  apply  it. 

These  design  requirements  were  documented  using  flowcharts 
and  wireframes  to  convey  the  navigation  and  site  architecture  for 
an  external  software  vendor  to  develop  the  LATIST  prototype. 
Additionally,  two  logical  data  models  were  developed  to  support 
the  dynamic  nature  of  the  “select  best  technology”  component  of 
LATIST  and  enable  a  searchable  repository  of  the  research.  Fur¬ 
thermore,  two  short  videos  were  designed  and  developed  by  a 
video  producer  to  introduce  users  to  the  purpose,  capabilities,  and 
navigation  of  LATIST.  Based  on  the  documented  requirements,  an 
initial  prototype  was  developed  to  begin  usability  testing. 

Usability  Testing 

Two  rounds  of  usability  testing  were  conducted  to  iteratively 
improve  the  prototype  based  on  expert  and  user  feedback.  Both 
rounds  were  intended  to  determine  design  inconsistencies  and 
usability  problems  to  establish  user  performance  and  user  satis¬ 
faction  levels.  Both  rounds  consisted  of  two  phases;  Phase  I  relied 
on  the  GMU  Immersion  team  and  proxy  participant  feedback  while 
Phase  II  relied  on  end-user  (DAU)  participant  feedback. 

In  Round  1,  the  usability  testing  focused  on  participant  percep¬ 
tions  and  sought  their  recommendations  for  improvement.  In  Phase 
I,  three  GMU  faculty  members  provided  expert  ISD  (Instructional 
System  Design)  review.  In  Phase  II,  DAU  stakeholder-users  con¬ 
ducted  an  asynchronous  review  of  the  prototype  and  answered  a 
brief  online  survey.  Survey  results  were  generally  positive  regard¬ 
ing  the  layout,  navigation,  and  overall  functionality  of  the  LATIST 
prototype.  Improvements  were  accomplished  in  preparation  for 
Round  2  usability  testing.  In  Round  2,  Phase  I  testing  again  relied 
on  GMU  faculty  members  to  provide  expert  ISD  review.  In  Phase  II, 
testing  took  place  at  DAU’s  Fort  Belvoir  campus  to  capture  repre¬ 
sentative  user  performance  and  user  satisfaction  under  controlled 
testing  conditions.  Nine  participants  selected  by  DAU  received  a 
brief  overview  of  LATIST  prior  to  the  usability  testing.  Once  the  test 
began,  participants  evaluated  LATIST  using  a  Web  browser  organic 
to  the  DAU  computer.  A  GMU  Immersion  team  member  observing 
the  test  guided  the  participants  to  the  LATIST  website.  Participants 
were  encouraged  to  think-aloud  as  they  used  LATIST  while  complet¬ 
ing  scenarios  provided  for  the  test.  Their  thoughts  were  captured 
using  a  tape  recorder,  and  in  some  cases  supplementary  notes  were 
taken.  Round  2  results  indicated  that  the  majority  of  participants 
regarded  LATIST  as  an  online  support  tool  that  would  be  helpful  in 
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raising  awareness  of  technology  options  among  DAU  staff  and  that, 
with  further  development,  LATIST  would  be  a  good  resource.  One 
participant  noted  that  LATIST  would  be  a  “good  idea  generator.” 
Recommendations  stated  independently  by  at  least  two  of  the  nine 
participants  included: 

•  Add  more  multimedia  resources  as  the  current  use  of 
videos  was  good; 

•  Add  more  examples  explaining  how  to  integrate  tech¬ 
nology  into  learning  assets; 

•  Add  a  means  to  easily  share  information  in  the  tool  such 
as  through  social  media  or  e-mail  options; 

•  Ensure  the  tool  is  scalable,  the  information  relevant,  and 
the  content  up-to-date;  and 

•  Provide  more  connectivity  across  components  and 
between  subcomponents  within  “select  best  technol¬ 
ogy”  component. 


LATIST  Prototype 

Based  on  the  usability  testing  results,  the  LATIST  prototype  was 
revised  to  include  the  main  features  and  functions  intended  to  sup¬ 
port  DAU  faculty  and  staff  in  integrating  ALT.  As  such,  LATIST  is  best 
described  through  its  main  navigation  pages:  LATIST  Home  Page, 
Explore  Research,  Select  Best  Technology,  and  Apply  Technology. 

The  LATIST  Home  Page  introduces  the  purpose  and  capabili¬ 
ties  of  LATIST,  providing  two  video  links  that  further  explain  what 
LATIST  is  and  how  to  use  it  (Figure  3).  LATIST  provides  many 
global  features.  Users  will  be  able  to  quickly  access  the  three  main 
components:  Explore  Research,  Select  Best  Technology,  and  Apply 
Technology  as  well  as  access  any  one  page  of  content  using  a  Tech¬ 
nology  quick  links  function.  In  the  future,  the  team  envisions  that 
users  will  be  able  to  log  in  to  add  personal  features  such  as  rating 
articles,  uploading  content,  and  taking  personal  notes.  An  advanced 
search  function  would  be  programmed  to  locate  all  resources 
based  on  filtering  agents  such  as  date,  title,  keyword,  and  author. 
A  dictionary  would  be  included  to  provide  quick  reference  on  what 
a  technology  is  and  define  the  influential  factors  significant  to  the 
selection  of  ALT.  A  “Help”  feature  would  target  technical  issues 
related  to  system  features  such  as  uploading  documents  to  LATIST. 

The  Explore  Research  component  of  LATIST  is  a  research-based 
body  of  knowledge  on  ALT  organized  into  three  broad  categories: 
(a)  Social  Media,  (b)  Virtual  Worlds/Games  and  Simulations,  and 
(c)  Mobile  Technologies  (Figure  4).  While  one  user  may  be  satisfied 
with  reviewing  a  technology  overview,  advantages/disadvantages, 
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FIGURE  3.  LATIST  HOME  PAGE 
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and  best  practices  of  a  specific  ALT,  another  may  want  to  pursue 
more  in-depth  research  by  reviewing  the  available  literature  of  that 
technology.  The  team  envisions  that  DAU  faculty  and  staff  will  be 
able  to  print,  share,  add,  upload,  mark  their  favorites,  rate  resources, 
and  select  articles  rated  highly  by  their  peers.  Additionally,  in  the 
future  the  system  will  provide  “Amazon-type”  recommendations  for 
other  resources  to  review  based  on  tagging  or  other  such  classifica¬ 
tion-type  metadata.  Users  will  be  able  to  easily  and  intuitively  move 
within  the  different  information  sections  of  the  Explore  Research 
component  and  across  all  LATIST  components. 

The  Select  Best  Technology  component  of  LATIST  guides  users 
to  make  informed  decisions  about  which  technologies  to  integrate 
into  learning  assets  in  a  pedagogically  sound  manner  while  taking 
DAU-specific  criteria  into  consideration.  This  component  has  two 
subcomponents:  the  Decision  Aide  and  the  Factors  Grid  (Figure  5). 
Through  the  Decision  Aide,  users  select  a  learning  objective  level 
that  matches  the  learning  objective  for  an  identified  DAU  course 
or  learning  asset.  The  Decision  Aide  responds  by  providing  a  list  of 
potential  instructional  strategies  for  that  learning  objective  level. 
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FIGURE  4.  EXPLORER  RESEARCH:  BLOGGING-OVERVIEW 


jeo 


-ft 


L  ATI  ST 


ILU1LJ  ii"  m  *^i 


•m  irtM* 


FIGURE  5.  SELECT  BEST  TECHNOLOGY  HOME  PAGE 


iLATl  ST 


rinin 

l#tl 


p«n  hrao| 


July  2011 


326 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


Based  on  the  user-selected  instructional  strategy,  the  system  returns 
a  “best  technology.”  The  Factors  Grid  allows  the  user  to  evaluate 
technologies  based  upon  criteria  specific  to  DAU  such  as  bandwidth, 
information  stability,  development  cost,  maintenance  cost,  and 
speed-to-market.  The  team  envisions  that  these  two  tools  would 
become  connected  or  intertwined  so  both  types  of  considerations 
(learning  objectives  and  influential  factors)  can  be  addressed  in  a 
singular  activity. 

The  Apply  Technology  component  enables  the  user  to  learn  how 
to  apply  a  specific  technology  by  providing  options  to  learn  how  to 
implement  it;  view  real  world  examples  of  use  in  a  DAU  context  and 
other  business,  military,  and  educational  contexts;  and  gain  hands- 
on  practice  (Figure  6).  The  user  can  access  the  information  through 
a  combination  of  embedded  or  hyperlinked  videos,  text  documents, 
or  URLs  to  external  websites.  The  information  provided  in  the  Apply 
Technology  component  will  allow  the  user  to  incorporate  a  selected 
technology  suitable  for  a  learning  asset. 

The  LATIST  Home  Page  introduces  LATIST  to  new  and 
occasional  users  while  the  three  components  of  LATIST  work  hand- 

FIGURE  6.  APPLY  TECHNOLOGY:  BLOGGING-HOW  TO 
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in-hand  to  facilitate  the  understanding,  selection,  and  integration  of 
ALT  by  DAU  faculty  and  staff  into  DAU’s  learning  assets. 


Conclusions 

DAU  is  committed  to  research  and  excellence  in  education  and 
training.  As  such,  a  partnership  between  DAU  and  GMU’s  Instruc¬ 
tional  Technology  Immersion  Program  was  formed  to  conduct 
research  on  ALT.  A  performance  analysis  revealed  that  DAU  faculty 
and  staff  could  benefit  from  a  tool  that  summarizes  research  on  ALT, 
guides  selection  of  ALT,  and  enables  implementation  of  ALT.  As  a 
result,  LATIST  was  designed  as  a  scalable  electronic  performance 
support  tool  that  allows  DAU  faculty  and  staff  to:  (a)  explore  what 
the  research  says  about  a  technology  such  as  advantages/disad¬ 
vantages;  (b)  select  a  best  technology  for  user  conditions  such  as 
learning  objectives  and  bandwidth  constraints;  and  (c)  review  and 
learn  how  to  apply  a  selected  technology.  A  usage-centered  design 
approach  was  used  to  develop  the  LATIST  prototype,  and  two 
rounds  of  usability  testing  were  conducted  to  iteratively  improve  its 
design  and  functionality.  The  results  revealed  that  overall,  LATIST 
was  perceived  by  DAU  stakeholders  and  participants  as  a  highly 
valued  performance  support  system.  Specifically,  LATIST  was  per¬ 
ceived  as:  (a)  useful  in  raising  awareness  of  technology  options 
among  DAU  faculty  and  staff,  (b)  a  “good  idea  generator,”  and  (c)  a 
“good  resource.”  Additionally,  participants  recommended  enhance¬ 
ments  to  increase  performance  and  user  satisfaction.  Implementing 
these  recommendations  and  fully  realizing  the  envisioned  function¬ 
ality  associated  within  each  LATIST  component  will  ensure  that 
DAU  faculty  and  staff  can  use  LATIST  to  make  research-driven  and 
pedagogically  sound  decisions  regarding  the  integration  of  ALT  into 
their  learning  assets  to  improve  their  effectiveness  and  efficiency, 
and  empower  the  Defense  Acquisition  Workforce  to  better  manage 
and  execute  job  performance.  So  how  will  you  choose  a  technology 
for  integration  into  a  learning  asset?  Check  out  LATIST  at  http:// 
cehd.gmu.edu/LATIST. 


July  2011 


328 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


Authors'  Note 

LATIST  was  designed  and  developed  in  academic  year  2009- 
2010  at  George  Mason  University  (GMU)  by  a  team  of  nine  graduate 
students  enrolled  in  GMU’s  Instructional  Technology  (IT)  Immersion 
Program  and  supervised  by  IT  faculty  Dr.  Nada  Dabbagh  and  Dr. 
Kevin  Clark.  The  Immersion  Program,  a  resident  graduate  program 
of  the  College  of  Education  of  Human  Development,  is  designed  to 
allow  teams  of  6-10  graduate  students  to  immerse  themselves  in 
project-based  learning  experiences  that  require  them  to  utilize  and 
apply  Instructional  Design  and  Development  (IDD)  principles  and 
processes  through  authentic  practice.  This  goal  is  achieved  through 
research  and  training  development  grants  that  engage  student 
teams  and  faculty  in  real-world  IDD  projects.  These  grants  enable 
the  integration  of  research,  theory,  and  practice  in  an  authentic 
problem-solving  context,  resulting  in  a  beneficial  situation  for  the 
funding  organization,  the  university,  the  program,  the  students,  and 
the  faculty.  For  more  information  about  the  immersion  program 
philosophy  and  projects,  visit  http://immersion.gmu.edu/.  To  learn 
more  about  this  DAU  research  project,  visit  http://immersion09. 
onmason.com/. 
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acquisition  professional.  The  reviews  should  be  400  words 
or  fewer,  describe  the  book  and  its  major  ideas,  and  explain  its 
relevance  to  defense  acquisition.  Please  send  your  reviews 
to  the  Managing  Editor,  Defense  Acquisition  Research  Journal: 
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Review: 

Megaprojects  have  some  distinct  characteristics:  Unusu¬ 
ally  large  by  definition,  they  require  significant  amounts  of 
capital  expenditure  (hundreds  of  millions  of  dollars)  and 
human  resources;  they  produce  complex  systems  with  high 
levels  of  technological  innovation;  and  they  have  the  potential 
to  change  their  surrounding  economic,  social,  and  organiza¬ 
tional  environment. 

It  is  through  projects  and  programs  that  assets  and 
knowledge  are  developed,  and  those  assets  and  infrastruc¬ 
ture  enable  the  necessary  operations  and  supply  chains. 
Clearly,  they  are  important  for  the  organizations,  individuals,  economy,  and 
society.  However,  it  is  also  clear  that  the  performance  record  of  megaprojects 
is  quite  poor,  with  significant  cost  and  schedule  overruns. 

The  authors  look  at  the  performance  record  of  megaprojects  from  around 
the  globe  though  a  risk  “lens.”  They  provide  an  in-depth  analysis  of  three 
transport  megaprojects:  the  Channel  Tunnel  between  the  United  Kingdom 
and  France;  the  Great  Belt  bridge/tunnel  in  Denmark;  and  the  0resund  Bridge 
between  Denmark  and  Sweden.  They  focus  on  the  front-end  part  of  the  projects, 
that  is,  the  feasibility/appraisal  stage.  These  detailed  analyses  are  complemented 
with  data  from  other  major  projects  in  both  the  public  and  private  sectors, 
including  the  transport,  information  technology,  oil  and  gas,  and  aerospace 
sectors. 

Through  their  analyses,  the  authors  critique  the  “conventional  approach 
to  megaproject  development”  (p.  86)  and  come  up  with  a  number  of  inter¬ 
esting  findings.  For  example,  “cost  estimates  used. ..in  decision  making. ..are 
systematically  and  significantly  deceptive”  (p.  20);  “over  optimistic  estimates 
of  project  viability  in  the  initial  planning  stage  and  inadequate  analysis  of  risk 
and  uncertainty”  (p.  41);  and  “accountability  is  low  for  parties  involved  in  project 
development  and  implementation”  (p.  45). 

The  main  reasons  for  poor  performance  identified  by  the  authors  include 
the  poor  consideration  of  risks;  institutional  issues  (such  as  lack  of  stakeholder 
involvement  or  a  lack  of  clearly  defined  roles);  and  a  lack  of  accountability  in  the 
project  decision-making  process.  The  authors  call,  perhaps  optimistically,  for  a 
mechanism  that  will  enforce  accountability  and  transparency.  To  this  end,  they 
provide  an  overview  of  a  number  of  instruments  that  might  help. 

This  book  is  of  interest  to  the  defense  acquisition  community,  in  part  because 
it  shows  that  overruns  are  not  limited  to  defense  projects  alone,  but  also 
because  it  argues  that  the  cause  of  cost  growth  is  not  due  solely  to  “unrealisti¬ 
cally  optimistic  estimates,”  as  Schwartz  (2010,  p.  16)  cites  in  a  recent  report  for 
Congress.  Rather,  there  are  other  dynamics  at  play  here  that  contribute  to  proj¬ 
ect  escalation:  organizational,  cultural,  behavioral,  and  cognitive.  It  is  through 
understanding  these  dynamics  that  the  problems  surrounding  project  escalation 
can  begin  to  be  addressed  more  effectively. 
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and  Librarians  (Rev.  Ed.),  Bethesda,  MD:  Congressional  Information  Service. 

Pages  should  be  double-spaced  and  organized  in  the  following  order:  title 
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